Don't we all love to execute scripts without the geeky looking terminal ?, say we have a shell or a python or a tcl-tk script ,we want now to run this as any app with an icon, Well there is a way.
Install gnome panel by using
$ sudo apt-get install gnome-panel
gnome-panel is just a taskbar application in desktop for ubuntu.
Once the installation is done type
$ gnome-desktop-item-edit --create-new ~/Desktop
you will then observe a create launcher applet , select type as Application
Give any funky name you want for your application.
And in the command field, provide the command which you use to run your script in the terminal.
so to run a tcl-tk application , use wish path_name/file_name.tcl
You can now click the icon to run your script.
You can also edit your application icon , just open it in gedit , the icon is saved as name.desktop
in the above case someapp.desktop
#!/usr/bin/env xdg-open
[Desktop Entry]
Version=1.0
Type=Application
Terminal=false
Icon[en_IN]=python
Exec=wish path_name/file_name.tcl
Name[en_IN]=someapp
Name=someapp
To all coders out there,a portal for collaboration,appreciation and merriment in the kingdom of codes !
Wednesday, 30 January 2013
extracting emotion from text
Here we shall look at a simple procedure to perform a sentiment analysis of the given text,
We proceed by making a list of words and associating with a value indicating the degree of emotion conveyed by the same.
something like :
awesome 4
bad -3
sad -3
cool 2
etc
Here the positive value indicate a positive emotion ,and the negative with negative emotion conveying words.The value indicate the degree of emotion conveyed by that word.
Now all we have to do is demarcate the text into constituent words and individually look up in this list,and sum up or take the average of the corresponding value to get the emotion value conveyed by the text.You can be wise enough to avoid usage of "stop words" in the list as they do not convey any meaning.
Personally i adore python for the awesome text processing capabilities , you can use the text as key and look it up as a dictionary ,have a look here.
A front end browser with a backend python script computing the emotion conveyed.
You can also use twitter to get general opinion on a particular topic
use http://search.twitter.com/search.json?q=topic_name
(all spaces in topic_name is to be replaced with +)
Twitter returns data in json format which you have to parse to extract the texts and use your python script to compute the emotion values which you can add or average as a whole to get general opinion
We proceed by making a list of words and associating with a value indicating the degree of emotion conveyed by the same.
something like :
awesome 4
bad -3
sad -3
cool 2
etc
Here the positive value indicate a positive emotion ,and the negative with negative emotion conveying words.The value indicate the degree of emotion conveyed by that word.
Now all we have to do is demarcate the text into constituent words and individually look up in this list,and sum up or take the average of the corresponding value to get the emotion value conveyed by the text.You can be wise enough to avoid usage of "stop words" in the list as they do not convey any meaning.
Personally i adore python for the awesome text processing capabilities , you can use the text as key and look it up as a dictionary ,have a look here.
A front end browser with a backend python script computing the emotion conveyed.
You can also use twitter to get general opinion on a particular topic
use http://search.twitter.com/search.json?q=topic_name
(all spaces in topic_name is to be replaced with +)
Twitter returns data in json format which you have to parse to extract the texts and use your python script to compute the emotion values which you can add or average as a whole to get general opinion
Using google map service without API key
To use google map service, you are instructed by the developer manual to generate an api key using your google account.You may have to cough up money to google if the number of hits to your site increase beyond a certain number.
But can we do the same sans api key ?+
Yes, well at-least for the geocoding and reverse geocoding service.
All you have to do is to ping : https://maps.google.com/maps?q="location name"
example https://maps.google.com/maps?q=washington or https://maps.google.com/maps?q=new+york
see how i replace spaces with + in the url.
To use it locally request data in json format, something like https://maps.google.com/maps?q="location name"&output=json
All you have to do is parse it for the required fields and there you have it,your data with no api key. Don't we love free stuff !!
But can we do the same sans api key ?
Yes, well at-least for the geocoding and reverse geocoding service.
All you have to do is to ping : https://maps.google.com/maps?q="location name"
example https://maps.google.com/maps?q=washington or https://maps.google.com/maps?q=new+york
see how i replace spaces with + in the url.
To use it locally request data in json format, something like https://maps.google.com/maps?q="location name"&output=json
All you have to do is parse it for the required fields and there you have it,your data with no api key. Don't we love free stuff !!
Wednesday, 12 December 2012
PHP FAQ
How to secure get parameters in php ?
Try using
sending end
$a="some crap";
$key = 'pass';// can use your own key here
$string = " ".$a." ";
$encrypteda = base64_encode(mcrypt_encrypt(MCRYPT_RIJNDAEL_256, md5($key), $string, MCRYPT_MODE_CBC, md5(md5($key))));
receiving end
$key = 'pass';
$rand=($_GET['parameter']);
$rand = str_replace(" ", "+", $rand);
$rand = rtrim(mcrypt_decrypt(MCRYPT_RIJNDAEL_256, md5($key), base64_decode($rand), MCRYPT_MODE_CBC, md5(md5($key))), "\0");
$rand = str_replace(" ", "", $rand);
--------------------------------------------------------------------------------------------------------------------
How to download multiple files in zipped form ?
Want to download files gmail style ? , well you can do it by using simple php script
we shall be using the following approach
open a ziparchive
add files to the ziparchive (you can add multiple files here )
close it after adding
Now just pass a header information containing the name of zip archive
finally read the archive file
That would be -
a reference to archive (inspired by java i guess!)
$ziparch= new ZipArchive()
open an archive
$ziparch->open($somename);
Then add files by
A loop
{
$ziparch->addFile($filename);
}
Close once done
$ziparch->close();
followed by provision of header information indicating the ziparchive name
header("Content-type: application/zip");
header("Content-Disposition: attachment; filename=$somename");
header("Pragma: no-cache");
header("Expires: 0");
finally
readfile("$somename");
--------------------------------------------------------------------------------------------------------------------
How to pass data between php pages ?
If you intend to pass data and yet find the popular post and get method to be incompatible use session .
all you got to do is start session at the beginning of the script
session_start()
Add data to session variables
(say in page 1)
$_SESSION['identifier']=$data;
Retrieve data by
(say in page 2)
$get_data=$_SESSION['identifier'];
--------------------------------------------------------------------------------------------------------------------
How to periodically execute php scripts ?
Without any user pinging the server, you can use cron service to schedule execution of scripts.Check out crontab in your favorite search engine.But there is a catch ,you have to incorporate this service onto the server and in some of the cases it is non trivial.
another method is to use the php's time() which will give the unix time stamp. well You can use just this.
Consider a time() instant 1358185093 ,get the ,say last 5 numbers or 85093
now in your index.php or in your php script use this
if($last_5_digit == 50000)
{
execute a periodic script.
}
NOTE : this works good for servers which get heavy traffic,since the users are constantly pinging the server (even when time()(last 5) == 5000)
The best ,easier method which i generally use is to add a time field in a table and observe the current time .Then delete all rows whose time field and the current time difference is beyond a threshold.
------------------------------------------------------------------------------------------------------------------------------
How to ping urls when your server is behind a proxy ?
use
$acontext = array(
'http' => array(
'proxy' => 'tcp://proxy_server:port', // example tcp://10.10.10.10//3128
'request_fulluri' => true,
),
);
$cxContext = stream_context_create($acontext);
$geturlcontents = file_get_contents("$url", False, $cxContext);
Incidentally for getting web resource in python
use
html=urllib2.urlopen('http://python.org/').read()
but ensure in case of proxy server,to set your environment variable http_proxy
use $ export http_proxy=http://proxy_server:port
$ echo $http_proxy should echo something like http://proxy_server:port/
Try using
sending end
$a="some crap";
$key = 'pass';// can use your own key here
$string = " ".$a." ";
$encrypteda = base64_encode(mcrypt_encrypt(MCRYPT_RIJNDAEL_256, md5($key), $string, MCRYPT_MODE_CBC, md5(md5($key))));
receiving end
$key = 'pass';
$rand=($_GET['parameter']);
$rand = str_replace(" ", "+", $rand);
$rand = rtrim(mcrypt_decrypt(MCRYPT_RIJNDAEL_256, md5($key), base64_decode($rand), MCRYPT_MODE_CBC, md5(md5($key))), "\0");
$rand = str_replace(" ", "", $rand);
--------------------------------------------------------------------------------------------------------------------
How to download multiple files in zipped form ?
Want to download files gmail style ? , well you can do it by using simple php script
we shall be using the following approach
open a ziparchive
add files to the ziparchive (you can add multiple files here )
close it after adding
Now just pass a header information containing the name of zip archive
finally read the archive file
That would be -
a reference to archive (inspired by java i guess!)
$ziparch= new ZipArchive()
open an archive
$ziparch->open($somename);
Then add files by
A loop
{
$ziparch->addFile($filename);
}
Close once done
$ziparch->close();
followed by provision of header information indicating the ziparchive name
header("Content-type: application/zip");
header("Content-Disposition: attachment; filename=$somename");
header("Pragma: no-cache");
header("Expires: 0");
finally
readfile("$somename");
--------------------------------------------------------------------------------------------------------------------
How to pass data between php pages ?
If you intend to pass data and yet find the popular post and get method to be incompatible use session .
all you got to do is start session at the beginning of the script
session_start()
Add data to session variables
(say in page 1)
$_SESSION['identifier']=$data;
Retrieve data by
(say in page 2)
$get_data=$_SESSION['identifier'];
--------------------------------------------------------------------------------------------------------------------
How to periodically execute php scripts ?
Without any user pinging the server, you can use cron service to schedule execution of scripts.Check out crontab in your favorite search engine.But there is a catch ,you have to incorporate this service onto the server and in some of the cases it is non trivial.
another method is to use the php's time() which will give the unix time stamp. well You can use just this.
Consider a time() instant 1358185093 ,get the ,say last 5 numbers or 85093
now in your index.php or in your php script use this
if($last_5_digit == 50000)
{
execute a periodic script.
}
NOTE : this works good for servers which get heavy traffic,since the users are constantly pinging the server (even when time()(last 5) == 5000)
The best ,easier method which i generally use is to add a time field in a table and observe the current time .Then delete all rows whose time field and the current time difference is beyond a threshold.
------------------------------------------------------------------------------------------------------------------------------
How to ping urls when your server is behind a proxy ?
use
$acontext = array(
'http' => array(
'proxy' => 'tcp://proxy_server:port', // example tcp://10.10.10.10//3128
'request_fulluri' => true,
),
);
$cxContext = stream_context_create($acontext);
$geturlcontents = file_get_contents("$url", False, $cxContext);
Incidentally for getting web resource in python
use
html=urllib2.urlopen('http://python.org/').read()
but ensure in case of proxy server,to set your environment variable http_proxy
use $ export http_proxy=http://proxy_server:port
$ echo $http_proxy should echo something like http://proxy_server:port/
Thursday, 6 December 2012
Document similarity using Hadoop
This article provides means to achieve document similarity which can be used for recommending similar webpages,news feeds,articles etc from the big web.
To begin with we shall be using hadoop for handling the colossal data.
The procedure involves considering each document and eliminating its stop words,(stop words are redundant words such as a,the,an,which etc..A list of stop words can be obtained from here) .Then we can use map reduce algorithm in hadoop distributed file system to identify the TFIDF associated with each word in a document
What is TFIDF ?
Well its composed of TF (Term frequency)and IDF(Inverse Document frequency)
The term frequency indicates the how many times a term(word ) occurs in a document.And the document frequency (DF in IDF, Also IDF=1/DF) indicates how many times the word occurs in the document.. However actually they give you the term count and document count for the word.To compute the frequency just divide the term count by total number of words in that document
(number of times W occurs in document D)
OR TF for word W= ------------------------------------------------------------------
(Total number of words in document D)
and IDF would be
(number of times W occurs in all documents)
DF = ------------------------------------------------------------------
(Total number of words in all documents)
So you can agree that higher the TF and lower the DF ,the word is important and can be used to analyze the characteristics of the document.Say a word w which occurs many times in a single document D but less in all other documents then this word can be used statistically to identify D.
So TFIDF would be TF * log(1/DF)
Log is logarithm to base 10
_
A_________
| w |
| w |
| w |
|_______w|
B__________
| w |
| |
| |
|_______|
C________
| w|
| |
| |
|_______|
D__________
| |
| |
| |
|_______|
Consider the above example where document A has a lot of W words the same W is rarely present in other document then it has a lower DF, and higher TF
Thus the TFIDF for the word W in document A is high.
So what we do here is identify the tfidf of all the word|document in corpus .Set a threshold for the TFIDF and consider those words whose tfidf is above the threshold.We can identify the cosine similarity between two documents from these TFIDF
Build a graph of the corpus wherin each node is a document and the edge between them are weighted based on cosine similarity .Higher the weight more similar are the nodes.
(Doc a)------<cosine similarity(a,b)>-----(Doc b) ..etc
Cosine-similarity between two documents a,b= sum ( TFIDF of a * TFIDF b)
Use graph clustering technique to cluster similar documents.We shall implement
this paper for clustering.
The technique involves randomly move about the graph initially and intialise the nodes by random labels , then we shall set a threshold on the weight of the edge between nodes , or we remove weak links.And we shall recursively analyze for a given node what label does majority of its neighbor are pointing to.We shall then assign this label to the node, we go on about like this until the labels assigned to the nodes don't change.
Most importantly this method automatically identifies the cluster size unlike k-means where we choose k. OR in this case k is identified by the algorithm not us.
Get the top tfidf keywords for the documents (using Hadoop)
Use majorclust to cluster the documents
When this is done use a php script or so. To use this data to provide the document/webpage relevance.
say the user chooses/views certain document/webpage the relevance can be provided to similar context
Here each document's name is a url as in url1,url2 etc
Get the tfidf hadoop implementation here, Edit this particular code
1. Add more stop words use this , Look at WordFrequenceInDocument class and observe googleStopwords.add("word");.. use the list of stopwords in the given file and copy it here.
2.Write the output to a file and parse it to obtain the file format as shown in the pics earlier so the python script can be run as such
Majorclust for clustering docs are implemented using this python script,(got it from stackoverflow response and edited as per requirement) see here
To illustrate the entire process:
And the architecture for implementation would be:
To begin with we shall be using hadoop for handling the colossal data.
The procedure involves considering each document and eliminating its stop words,(stop words are redundant words such as a,the,an,which etc..A list of stop words can be obtained from here) .Then we can use map reduce algorithm in hadoop distributed file system to identify the TFIDF associated with each word in a document
What is TFIDF ?
Well its composed of TF (Term frequency)and IDF(Inverse Document frequency)
The term frequency indicates the how many times a term(word ) occurs in a document.And the document frequency (DF in IDF, Also IDF=1/DF) indicates how many times the word occurs in the document.. However actually they give you the term count and document count for the word.To compute the frequency just divide the term count by total number of words in that document
(number of times W occurs in document D)
OR TF for word W= ------------------------------------------------------------------
(Total number of words in document D)
and IDF would be
(number of times W occurs in all documents)
DF = ------------------------------------------------------------------
(Total number of words in all documents)
So you can agree that higher the TF and lower the DF ,the word is important and can be used to analyze the characteristics of the document.Say a word w which occurs many times in a single document D but less in all other documents then this word can be used statistically to identify D.
So TFIDF would be TF * log(1/DF)
Log is logarithm to base 10
_
A_________
| w |
| w |
| w |
|_______w|
B__________
| w |
| |
| |
|_______|
C________
| w|
| |
| |
|_______|
D__________
| |
| |
| |
|_______|
Consider the above example where document A has a lot of W words the same W is rarely present in other document then it has a lower DF, and higher TF
Thus the TFIDF for the word W in document A is high.
So what we do here is identify the tfidf of all the word|document in corpus .Set a threshold for the TFIDF and consider those words whose tfidf is above the threshold.We can identify the cosine similarity between two documents from these TFIDF
Build a graph of the corpus wherin each node is a document and the edge between them are weighted based on cosine similarity .Higher the weight more similar are the nodes.
(Doc a)------<cosine similarity(a,b)>-----(Doc b) ..etc
Cosine-similarity between two documents a,b= sum ( TFIDF of a * TFIDF b)
Use graph clustering technique to cluster similar documents.We shall implement
this paper for clustering.
The technique involves randomly move about the graph initially and intialise the nodes by random labels , then we shall set a threshold on the weight of the edge between nodes , or we remove weak links.And we shall recursively analyze for a given node what label does majority of its neighbor are pointing to.We shall then assign this label to the node, we go on about like this until the labels assigned to the nodes don't change.
Most importantly this method automatically identifies the cluster size unlike k-means where we choose k. OR in this case k is identified by the algorithm not us.
Get the top tfidf keywords for the documents (using Hadoop)
Use majorclust to cluster the documents
When this is done use a php script or so. To use this data to provide the document/webpage relevance.
Here each document's name is a url as in url1,url2 etc
Get the tfidf hadoop implementation here, Edit this particular code
1. Add more stop words use this , Look at WordFrequenceInDocument class and observe googleStopwords.add("word");.. use the list of stopwords in the given file and copy it here.
2.Write the output to a file and parse it to obtain the file format as shown in the pics earlier so the python script can be run as such
Majorclust for clustering docs are implemented using this python script,(got it from stackoverflow response and edited as per requirement) see here
To illustrate the entire process:
And the architecture for implementation would be:
Subscribe to:
Posts (Atom)







