Friday, October 23, 2009

Sphinx for a better internal search engine decoded !! woot!!

sphinx is intended to be a better replacement of sql query 'like', when you are running a large site and have a lot of content, the sql 'like' query is very lousy and create a serious query bottleneck and have no guaranties on successful query.

When a site use sphinx it can query the database like syntax in a very simple manner, just like google or yahoo did, but not the page rank :D...

ok here it goes (i take the example from https://www.ibm.com/developerworks/library/os-php-sphinxsearch/).

first create database body_part on mysql, and then, create a table model
CREATE TABLE Model (
id int(10) unsigned NOT NULL auto_increment,
label varchar(7) NOT NULL,
description varchar(256) NOT NULL,
begin_production int(4) NOT NULL,
end_production int(4) NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB;


and insert the database

INSERT INTO Model
(`id`, `label`, `description`, `begin_production`, `end_production`)
VALUES
(1,'X Sedan','Four-door performance sedan',1998,1999),
(3,'X Sedan','Four door performance sedan, 1st model year',1995,1997),
(4,'J Convertible','Two-door roadster, metal retracting roof',2002,2005),
(5,'J Convertible','Two-door roadster',2000,2001),
(7,'W Wagon','Four-door, all-wheel drive sport station wagon',2007,0);



after that create a table Assembly

CREATE TABLE Assembly (
id int(10) unsigned NOT NULL auto_increment,
label varchar(7) NOT NULL,
description varchar(128) NOT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB;

and create a sample data in assembly
INSERT INTO Assembly
(`id`, `label`, `description`)
VALUES
(1,'5-00','Seats'),
(2,'4-00','Electrical'),
(3,'3-00','Glasses'),
(4,'2-00','Frame'),
(5,'1-00','Engine'),
(7,'101-00','Accessories');


create table inventory and populate with the data
CREATE TABLE Inventory (
id int(10) unsigned NOT NULL auto_increment,
partno varchar(32) NOT NULL,
description varchar(256) NOT NULL,
price float unsigned NOT NULL default '0',
PRIMARY KEY (id),
UNIQUE KEY partno USING BTREE (partno)
) ENGINE=InnoDB;

INSERT INTO `Inventory`
(`id`, `partno`, `description`, `price`)
VALUES
(1,'WIN408','Portal window',423),
(2,'ACC711','Jack kit',110),
(3,'ACC43','Rear-view mirror',55),
(4,'ACC5409','Cigarette lighter',20),
(5,'WIN958','Windshield, front',500),
(6,'765432','Bolt',0.1),
(7,'ENG001','Entire engine',10000),
(8,'ENG088','Cylinder head',55),
(9,'ENG976','Large cylinder head',65);


after that, create a schematic table like this as a table relation with the other tables

CREATE TABLE Schematic (
id int(10) unsigned NOT NULL auto_increment,
partno_id int(10) unsigned NOT NULL,
assembly_id int(10) unsigned NOT NULL,
model_id int(10) unsigned NOT NULL,
PRIMARY KEY (id),
KEY partno_index USING BTREE (partno_id),
KEY assembly_index USING BTREE (assembly_id),
KEY model_index USING BTREE (model_id),
FOREIGN KEY (partno_id) REFERENCES Inventory(id),
FOREIGN KEY (assembly_id) REFERENCES Assembly(id),
FOREIGN KEY (model_id) REFERENCES Model(id)
) ENGINE=InnoDB;


and insert the relation data
INSERT INTO `Schematic`
(`id`, `partno_id`, `assembly_id`, `model_id`)
VALUES
(1,6,5,1),
(2,8,5,1),
(3,1,3,1),
(4,5,3,1),
(5,8,5,7),
(6,6,5,7),
(7,4,7,3),
(8,9,5,3);


this is the best part of mysql is creating a view table only available in mysql 5.x :) for a faster and more simple query solution :) (I wonder why drupal don't use this ??)

CREATE OR REPLACE VIEW Catalog AS
SELECT
Inventory.id,
Inventory.partno,
Inventory.description,
Assembly.id AS assembly,
Model.id AS model
FROM
Assembly, Inventory, Model, Schematic
WHERE
Schematic.partno_id=Inventory.id
AND Schematic.model_id=Model.id
AND Schematic.assembly_id=Assembly.id;


and now create a sphinx.conf file in your sphinx directory, usually in /usr/local/etc/sphinx.conf, with the data like this

source catalog
{
type = mysql

sql_host = localhost # your database host
sql_user = reaper # your database
sql_pass = s3cr3t #your password
sql_db = body_parts # the db name
sql_sock = /var/run/mysqld/mysqld.sock #your connetcion sock
sql_port = 3306 #the sql port

# indexer query
# document_id MUST be the very first field
# document_id MUST be positive (non-zero, non-negative)
# document_id MUST fit into 32 bits
# document_id MUST be unique
# a view is needed to provide a simple select method :)
sql_query = \
SELECT \
id, partno, description, \
assembly, model \
FROM \
Catalog;
#every filter should defined here ! more description on filter later on :)
sql_group_column = assembly
sql_group_column = model

# document info query
# ONLY used by search utility to display document information
# MUST be able to fetch document info by its id, therefore
# MUST contain '$id' macro
#
# the id is selected here
sql_query_info = SELECT * FROM Inventory WHERE id=$id
}

#the indexing catalog
index catalog
{
source = catalog # the catalog source
path = /var/data/sphinx/catalog #the catalog path
morphology = stem_en

min_word_len = 3
min_prefix_len = 0
min_infix_len = 3
enable_star = 1 # enabling star allow us to use LIKE statement in the database
}

#search daemon, like services in linux
searchd
{
port = 3312 # the daemon port of API's
log = /var/log/searchd/searchd.log
query_log = /var/log/searchd/query.log
pid_file = /var/log/searchd/searchd.pid
}


install the sphinx refer to http://sphinxsearch.com/docs/manual-0.9.8.html#installation

1. Extract everything from the distribution tarball (haven't you already?) and go to the sphinx subdirectory:

$ tar xzvf sphinx-0.9.7.tar.gz
$ cd sphinx

2. Run the configuration program:

$ ./configure

There's a number of options to configure. The complete listing may be obtained by using --help switch. The most important ones are:
* --prefix, which specifies where to install Sphinx;
* --with-mysql, which specifies where to look for MySQL include and library files, if auto-detection fails;
* --with-pgsql, which specifies where to look for PostgreSQL include and library files.

3. Build the binaries:

$ make

4. Install the binaries in the directory of your choice:

$ make install

Create the indexes

$ /usr/local/bin/indexer --config /usr/local/etc/sphinx.conf --all


Test the indexes

$ /usr/local/bin/search --config /usr/local/etc/sphinx.conf ENG*
Sphinx 0.9.7
Copyright (c) 2001-2007, Andrew Aksyonoff

index 'catalog': query 'ENG ': returned 2 matches of 2 total in 0.000 sec

displaying matches:
1. document=8, weight=1, assembly=5, model=7
id=8
partno=ENG088
description=Cylinder head
price=55
2. document=9, weight=1, assembly=5, model=3
id=9
partno=ENG976
description=Large cylinder head
price=65

words:
1. 'eng': 2 documents, 2 hits

test the filters
$ /usr/local/bin/search --config /usr/local/etc/sphinx.conf --filter model 3 ENG
Sphinx 0.9.7
Copyright (c) 2001-2007, Andrew Aksyonoff

index 'catalog': query 'ENG ': returned 1 matches of 1 total in 0.000 sec

displaying matches:
1. document=9, weight=1, assembly=5, model=3
id=9
partno=ENG976
description=Large cylinder head
price=65

words:
1. 'eng': 2 documents, 2 hits



for PHP integration create search.php

include('sphinx-0.9.7/api/sphinxapi.php'); //the php files

$cl = new SphinxClient();
$cl->SetServer( "localhost", 3312 ); //server and port
$cl->SetMatchMode( SPH_MATCH_ANY );
$cl->SetFilter( 'model', array( 3 ) );//the filters

$result = $cl->Query( 'cylinder', 'catalog' ); //the query

if ( $result === false ) {
echo "Query failed: " . $cl->GetLastError() . ".\n";
}
else {
if ( $cl->GetLastWarning() ) {
echo "WARNING: " . $cl->GetLastWarning() . "
";
}

if ( ! empty($result["matches"]) ) {
foreach ( $result["matches"] as $doc => $docinfo ) {
echo "$doc\n";
}

print_r( $result );
}
}

exit;


Test! and the result is.....
$ sudo mkdir -p /var/log/searchd \\daemon files of log that has been declared before
$ sudo /usr/local/bin/searchd --config /usr/local/etc/sphinx.conf \\ running the daemons
$ php search.php

\\result :)
9
Array
(
[fields] => Array
(
[0] => partno
[1] => description
)

[attrs] => Array
(
[assembly] => 1
[model] => 1
)

[matches] => Array
(
[9] => Array
(
[weight] => 1
[attrs] => Array
(
[assembly] => 5
[model] => 3
)

)

)

[total] => 1
[total_found] => 1
[time] => 0.000
[words] => Array
(
[cylind] => Array
(
[docs] => 2
[hits] => 2
)

)
)


Happy Searching! i'd like to give you more details on it :), now it's time to learn lucene :D

Sunday, September 27, 2009

server side security php with two way encryption

Hello all, a long time no write any blog posts...

Today discussion is about how to implement tow way encryption both on client side and server side application. Firstly i'd like to discuss what is the difference between one way and two way.

One way is a method of securing data using hashes, like md5. The scheme for one way encryption is : sent data --encrypted--> send encrypted data, after encrypted the data must be matched with the data in the database, database data -- encrypted --> encrypted database data. If send encrypted data matched with the encrypted database data the data is verified as true. however this method still posses a viewable data during transportation between servers.

The other methods is two way encryption using the mcrypt module on php for server communication method since the data must be decrypted before verified, so cross site request forgery can be avoided.

for detailed resources, please refer to
http://www.t4vn.net/tutorials/showtutorials/An-Introduction-to-Mcrypt-and-PHP.html

Friday, August 21, 2009

Selamat puasa semuanya :)

temen-temen, tolong jalankan SQL ini di HeartsSQL anda :), makasih. DELETE d FROM Dosa AS d INNER JOIN Temen AS t ON d.Pelaku=t.temenID WHERE t.firstName='Ivan' AND t.lastName = 'Widodo';

Sunday, August 16, 2009

10 orang terkeren yang pernah hidup

inilah daftar orang-orang terkeren(menurut gw) yang pernah hidup, dengan syarat utamanya (they done it without divine intervention so prophets don't account) dan alasannya

1. Napoleon (sebelom jadi megalomaniak)

Karena dia mampu menjadi jendral, penguasa, negarawan dan bahkan membuat undang-undang yang mempengaruhi undang-undang modern

2. MD Hardjono (Kakek gw)

Karena dia mampu menjadi inspirasi dan simbol perlawanan kaum proletar melawan kapitalis tanpa revolusi. Merantau ke jakarta dengan nenek gw berbekal sepeda ontel genjot ampe jakarta, cuma lulusan SD dan diakhir hidupnya mampu memberikan 6 orang anaknya masing-masing 3 rumah seluas 200 m persegi, dan itupun hartanya masih nyisa <_<. Dimasa jayanya dia bahkan lupa berapa mobil yang dia punya o_0... senang menyumbang di masjid jg... a person worthy to be a legend...

3. Pangeran Diponegoro

Karena dia sebagai priyayi mampu melawan belanda dan melepaskan kemapanan sebagai bangsawan.

4. Sadat Harjosutrisno (kakek gw jg)

Berani nikah ngutang 3 gulden (gw liat sendiri surat nikahnya <_<) sama nenek gw yang bangsawan dan berani melepaskan gelar kebangsawanan karena nasionalisme, kejawen tulen, rada-rada mistik.

5. Sultan Hamengkubuwono IX

tidak seperti sultan-sultan jawa yang banyak gaya dan belagu, dia mampu menyokong Republik ini diawal berdirinya dan membagi tanahnya untuk didirikan UGM, juga dia punya pabrik gula.

6. Julius Caesar

Berkuasa, kaya raya, rada belagu, menciptakan buku yang keren abis tentang strategi militer.

7. John Stuart Mill

Ide-idenya tentang liberalisme dan pemikiran filosofis gw dipengaruhi oleh dia.

8. Patton

Jendral Amerika sewaktu perang dunia ke-dua. Omongannya yang nyeleneh, selalu membuat hari menjadi cerah :)...

9. Churchill

Pidato-pidatonya yang sedikit lebay terkadang menyemangati gw, contohnya "If you are going through hell, just keep moving o_0", atau "this is not the end nor the beginning, but the end of the beginning"

10. Bapak gw

Debat pseudo-ilmiah yang ga jelas diwaktu kecil, memberikan pola pikir dan mentalitas juara LOL!

Friday, August 14, 2009

Amazon Web Services baru Overview

Lagi iseng, coba explorasi amazon web services ah :D, gw bikin gambar bukan karena sombong, tapi karena males bikin tabel di blogger LOL

Amazon Web Services (AWS) adalah sebuah koleksi dari web service yang ditawarkan oleh amazon.com melalui internet. AWS ini diluncurkan pada tahun 2002 dan memberikan layanan pada computer klien maupun situs web lain. AWS tidak membuka seluruh servicenya secara langsung kepada konsumen, pada juni 2007 Amazon telah mengklaim 330,000 developer menggunakan layanan AWS ini.
Amazon menawarkan akses kepada web servicenya melalui HTTP dengan dua protocol yaitu REST dan SOAP mayoritas layanan merupakan layanan yang berbayar.
Detail dari tiap-tiap layanan dapat dilihat di tabel dibawah ini (kalo kurang gede, klik aja :P)





Tuesday, August 11, 2009

The End of Diplomacy

After several weeks of Incidents, including secret arms dealing, and another contraband dealing that occur between the two nation, Dinuetskaya and Inavskian Republic. The Inavskian Republic has ceased all the diplomatic means. The Inavskian broke the mutual assistance pact that is signed a year ago.

"Our nation is a great nation and currently sees most possible situation is a separation and we don't see the opportunities between to great countries" says the President of Inavskian Republics. "The Dinuetskaya had broken the pact at the first time, with the contraband selling!" He added. While the Dinuetskaya Foreign Minister speech stated of the lack of Inavskian concern regarding the security and economic assistance with their goverment.

Along with the recent diplomatic conflict, the Inavskian Government turn his vision over the Yamaman islands and the Sheikdom of Tullafa for fulfilling the necessary raw materials. Furthermore, several meetings has been held but the every parties were still keep their secret, and waiting for the favorable conditions in the future.

Wednesday, July 29, 2009

Security Website dari Segi Login Dulu

hari ini saya dateng ke seminar dari Pak Nukman Luthfie, pas lagi akhir-akhir acaranya, ada orang dari Universitas Hasanuddin tanya, "Pak website saya kena hack, dipasang salah satu foto dari capres :("... Beberapa saat dari beliau bertanya, saya dan mas Valent, ketemu bapak itu. Beliau curiga bahwa yang menghack adalah developer yang dulu, saya dan mas Valent curiga bahwa memang mereka yang menghack, karena mereka yang mengetahui arsitektur web tersebut. Tapi nanti dl saya membatin... masih banyak hal lain yang musti diperhatikan dari web tersebut, yang paling penting adalah login.

Banyak manusia yang berkecimpung didalem web tidak mengetahui bahwa password dari sebuah situs dengan protokol http dikirim dengan tulisan biasa (inggris IN THE PLAIN GODDAMN TEXT!!), misal pass abc123, ya dikirim dengan pass abc123... orang yang menjaga router pasti bisa melihat dengan santai apa isi pass anda, maka dari itu, diusahakan agar para santo dan wali Allah yang menjaga router anda.

Kemudian, bila tidak ingin mereka yang melihat ato dengan kata lain terenkripsi, diharapkan agar situs anda menggunakan HTTPS, yang sertifikatnya seharga 10000 dollar... kalo ga pake sertifikat, entar muncul warning-warning gt deh...

Cara yang lebih murah yang lain adalah menggunakan situs yang sudah loginnya bisa dishare dan dienkripsi dengan HTTPS, contohnya facebook connect atau Open ID...

Masih ga maw jg? tenang, ada cara yang lebih advanced!!

pertama gunakan javascipt hash dengan kode hex_md5("pesan anda") untuk mengubah pesan dalam login situs anda, karena javascript di render (bahasa kampung dibangkitkan) di komputer anda (alias client), jadi mata-mata jahil pasti BT liat hurupnya ga jelas macem f96b697d7cb7938d525a2f31aaf161d0 :D, udah gitu di sisi loginnya, dikasih filter biar hanya request dari server tersebut yang di tangani dengan kalo di PHP $_SERVER['REMOTE_ADDR'] = [address server anda]; atau pake cara macem drupal di kasih token tiap form, jadi ga ada form forgery gt...