Some Python String Manipulation Functions (Quick Reference)

Just putting some python string manipulation techniques in a place as quick reference

Suppose

a = “a quick brown fox run over the lazy dog”

 

reverse string

a[::-1]

 

last character of the string

a[-1:]

 

first 4 character of the string

a[:4]

 

split a string with white spaces

a.split()

 

split a string with character or string (not regular expression)

a.split(‘fox’)

 

find a string inside a string

a.find(‘run’)

a.find(‘run’,0)

a.find(‘run’,0,len(a))

 

find from end of the string

a.rfind(‘run’)

a.rfind(‘run’,0)

a.rfind(‘run’,0,len(a))

 

count number of occurrence in a string

a.count(‘a’)

 

join several string together with separators

st = “,”

seq = (“cat”, “tiger”, “lion”)

print st.join( seq )

 

change the case into opposite one, if lower go to upper, if upper go lower.

a. swapcase()

 

replace part of the string

a.replace(‘run’,’walk’)

 

 

left, right and center justify

a.ljust(10,’’)

a.rjust(10,’ ’)

a.center(10,’ ‘)

 

Speedup Your Database Using Efficient Design

Here are some tips where you have to concentrate on to improve performance during database design time. It’s not whole database design but will be some tricks to improve efficiency.

• Don’t afraid of normalization which mean don’t avoid join or try to avoid join. Joining is more efficient then denormalized.
• Try to avoid Boolean flag.
• Use Index cleverly (which fields will be used in search and ordering).
• Try to avoid index on string columns.
• Don’t index everything.
• Do not duplicate indexes.
• Be careful of redundant columns in an index or across indexes.
• Normalize first, and denormalized where appropriate.
• A NULL data type can take more room to store than NOT NULL.
• Choose appropriate character sets & collations — UTF16 will store each character in 2 bytes, whether it needs it or not, latin1 is faster than UTF8.
• Trigger is expensive so use it wisely.
• Be able to change your schema without ruining functionality of your code.
• As your data grows, indexing may change (cardinality and selectivity change). Structuring may want to change. Make your schema as modular as your code. Make your code able to scale. Plan and embrace change, and get developers to do the same.

 

How to Increase SQL Performance (Query Optimization)

Database and SQL performance largely depends on how one designing his database, system configuration, frequent database access type, load balancing etc. Though different database behave differently on a specific query because of their architectural difference, most of them have some identical performance increase tips. It’s basically related to SQL not on specific DB server. Here, I am trying to list some sql performance increase (query optimization) tips which I usually try to follow during SQL programming. One might not fulfill the entire requirement all the time but try to follow as much as possible.

 Try To Avoid:

• Don’t use DISTINCT in GROUP BY clause
• Don’t use ORDER BY in random number with too many data.
• Don’t put ORDER BY in text/blobs/binary data
• Try to avoid wildcards at the start or both ends of LIKE queries. It’s better to avoid LIKE queries.
• Try to avoid IN or NOT IN, <>. Actually IN or NOT in works like OR operation.
• Separate text/blobs/binary from metadata; don’t put text/blobs/binary in results if you don’t need them.
• Don’t SELECT/UPDATE unnecessary data during operation, use filter accurately.
• Avoid correlated subqueries in SELECT and WHERE clause.

 Try to Do:

• Always try to use column(s) name in select queries instead of picking all columns using asterisks (*). Try to select as less columns as possible. But you may do calculations in select operation. This may be faster instead of doing it in application layer.
• Try to use ORDER BY in index columns.
• Always try to put index columns first in the WHERE and ON clause
• Always use COUNT(*) instead of COUNT(column name). because when you put column name it checks for null and if it find null on that column it won’t count that row. btw, if you want specific column count then you may use it.
• Use UNION operations instead of OR operations
• Try to DELETE small amount of data at a time.
• Always try to use join instead of multiple queries or loops.
• Use groupwise maximum instead of subqueries.
• Try to split a complex query and join smaller ones when necessery.
• For better performance, Batch INSERT and REPLACE or use LOAD DATA instead of INSERT.

 

Get All Links From text/html Data Using Regular Expression (Link Extractor)

Sometime one might just want to extract all URLs from html or other text. It’s very easy to do that using regular expression. You may find a lots of regular expression to extract links from text. But I found this one ( \\(?\\b(https??://|www[.])[-A-Za-z0-9+&@#/%?=~_()|!:,.;]*[-A-Za-z0-9+&@#/%=~_()|] ) very useful, and it provided all links that I wanted. There may be some exception, but you need to found out that by using it. I hope it will give you 99% time all type of links.

Bellow you will find a java method which extract links and then put into a HashSet. This method gets an argument which will contain all text

private HashSet getAllLinks(String data){

 HashSet crawledUrlList = new HashSet();

        String lowValue = new String(data);       

        Pattern pattern = Pattern.compile(“\\(?\\b(https??://|www[.])[-A-Za-z0-9+&@#/%?=~_()|!:,.;]*[-A-Za-z0-9+&@#/%=~_()|]”);

        Matcher matcher = pattern.matcher(lowValue);

        while(matcher.find()){

            String url = matcher.group();

            crawledUrlList.add(url);

        }

return crawledUrlList;

    }

 

How to create a crawler using java !!

There are two ways to crawl web pages in java.

Most primitive but original way is that to open a socket in 80 no port and then use get statement to obtain content. It works almost like telnet.

• telnet google.com 80
• GET /  HTTP/1.0  (two new line)

By this way you will get content of the home page. Look at the sample method how can we do the whole procedure in java

public String urlCrawle(String url){

        this.insertCrawledUrl(url);       

        StringBuffer objBuffer = new StringBuffer(“”);

        try{

           URL objURL = new URL(url);

            String host = objURL.getHost();

            String path = objURL.getPath();

            if(path.length() == 0){

                path=”/”;

            }

            String outQuery = “GET “+path+”?”+objURL.getQuery()+” HTTP/1.0\n”;

            //System.out.println(outQuery);

            Socket s = new Socket(InetAddress.getByName(host), 80);

            PrintWriter out = new PrintWriter(new OutputStreamWriter(s.getOutputStream()));

            out.println(outQuery);

            out.flush();

            BufferedReader instream = new BufferedReader(new InputStreamReader(s.getInputStream()));

            String line = instream.readLine();

            if(line.contains(“HTTP/1.0 200”) || line.contains(“HTTP/1.1 200”)){

                while(line != null) {

                    objBuffer.append(line+”\n”);

                    line = instream.readLine();

                }

                s.close();

            }

        }

        catch(Exception ex){}

        //return this.stripTagFromHtml(objBuffer.toString());

        System.out.println(objBuffer.toString());

        return objBuffer.toString();

    }

The problem of this procedure is that you have to separate hostname, path and query string and then work with them individually. And it’s quite childish to work such a way in java as this is a very dynamic and high level language. But for education purpose it’s the most ultimate way to know the underline working procedure of the system.

As java has a very wide range of network programming library, one can use URL class to do web crawling and it’s the easiest and also effective way for web crawling. You may find an example method bellow.

public String urlCrawle(String url){

        this.insertCrawledUrl(url);

        StringBuffer objBuffer = new StringBuffer();

        try{

            URL hp = new URL(url);

            URLConnection hpCon = hp.openConnection();

            int len = hpCon.getContentLength();

            String line = “”;

            if(len>0){               

                BufferedReader instream = new BufferedReader(new InputStreamReader(hpCon.getInputStream()));

                line = instream.readLine();

                while(line != null) {

                    objBuffer.append(line+”\n”);

                    line = instream.readLine();

                }               

            }

        }

        catch(Exception ex){}

        //return this.stripTagFromHtml(objBuffer.toString());

        return objBuffer.toString();

    }