profeshunl newbie » Code

MongoDB as in huMONGOus, not retarded

william — Mon, 12 Apr 2010 11:01:43 +0000

Unlike the rest of the known world, I won’t write anything on the iPad. Yet. Instead, I’m still in the process of writing the upcoming parts of my Popurls Clone series and as per usual, it takes a lot more time than expected. In combination with a massive amount of bills, life is good! As it turns out, you actually need to “work” in order to get “money”. How, when and why did we agree upon that? That’s just as stupid as, say, doing a fully animated Star Wars prequel/sequel to take place during the Clone Wars. Oh, wait… While pondering life’s great mysteries, such as how you can evade paying your bills, I felt it was time to write something about databases. You see the connection, right? Good, because I don’t. A quick Google search for MySQL rendered roughly 117 million results so I figured that another article about it would be kinda superfluous. A search for MongoDB on the other hand resulted in “only” 1,2 million results. Much better. That means that if and when this article gets indexed by the Google, it will constitute no less than ~ 8.33 × 10^-7 % of the relevant search results, in stark contrast to the ~ 5.88 × 10^-9 %, should I’ve written some stuff about MySQL. Goodie! In short, this article will cover MongoDB and how you can use it with PHP. Good times ahead so if you’re a killjoy, you’re not welcome.

A brief introduction to MongoDB

First of all, MongoDB is not a relational database management system (RDBMS) such as MySQL. A relational DBMS is based on a relational model, based on first-order predicate logic as proposed by Edgar Frank Codd in 1969. Data and its internal relationships are more often than not stored as tables. These tables are what constitute the relations. The tables are organized using a kind of model of vertical columns and rows where there are a specified number of columns but the number of rows are, in theory, infinite. The data is fetched via queries using SQL. But enough about that – we’re here to talk about MongoDB which is a part of the NoSQL movement. In NoSQL there are basically three kinds of databases; column oriented, key-value pairs and document oriented. MongoDB employs the latter kind. A document in this context is a structure of data with a given number of properties. These properties can be strings, numbers, arrays or objects, etc. If you’ve dabbled with associative arrays or objects in PHP, you’ll know what this means. You can also group documents in collections. Also, there are something called sub-documents which are pretty much what you’d think they’d be.

In the case of MongoDB, we have no schema to play with but the power lies in the fact that we get to play with JSON-like structures. And JSON is good, right? MongoDB is written in C++, it just hit version 1.4.0 and it stores data using BSON, which is a “binary-encoded serialization of JSON-like documents”. Thankfully, there’s a native PHP extension. So the first thing you’d want to do is install it. Download the appropriate package and perhaps read the Quickstart. Assuming you use PECL, which you should, you can install it easily with the following command;

pecl install mongo

or you can do it manually by downloading the source

phpize
./configure --enable-mongo
make install

The install should have created an extension for you and put it in the extension directory. You may need to edit your PHP.ini and add

extension=mongo.so

where appropriate (hint: maybe where the other extensions are defined). If you get an error in the lines of “phpize: comand not found”, this page is for you. If you’re on linux, a quick fix is to install php5-dev with apt-get or php-devel with yum. Restart your webserver. Do it. If you’re interested in MongoDB but loathe PHP (god knows you have your reasons), there are drivers for other languages, such as C# and .NET (why are you reading this?), Clojure (you pervert), ColdFusion (oh no, you dideeent), Python (you’re all right) or Ruby (pff).

I feel the vibe

Let’s do this. Connect to your brand spanking new Mongo database at localhost (default port is 27017)

$the_mongo_connection = new Mongo();

Or if you’d like to connect to a remote host

$the_mongo_connection = new Mongo("10.0.0.2[:optional_port]");

Pro tip: use a persistent connection. New connections to the database can be very slow. Let this quote borrowed from php.net serve as a cautionary, um, well, example;

for ($i=0; $i<1000; $i++) {
    $m = new Mongo();
}

This will take roughly 18 seconds to execute. Yes, 18. That’s like an eternity on the Internets. If we use a persistent connection like so;

for ($i=0; $i<1000; $i++) {
    $m = new Mongo("localhost:27017", array("persist" => "x"));
}

…we’re down to 0.02 seconds. So, yeah, persistent may be the way to go. And by “may” I mean “do it, ass hat”. If you’re wondering about the “x” in the last piece of code, it’s because persistent connections need a unique identifier string.

The name of the database can use almost any character in the ASCII range, except a blank space, a period or just an empty string. You can actually name a database “null” if you’d like. Might be fun just to piss people off. Selecting a database can be done in two ways and isn’t harder than

$db = $the_mongo_connection->name_of_database;
$db = $the_mongo_connection->selectDB('name_of_database');

The latter is to be preferred if you’re using wonky characters in your database name, such as a comma, a slash or something like that. Valid, but it’ll break your code into millions and millions of pieces if you don’t use the quoted selectDB(). Now we need to select collection to work with. You might say that this is similar to selecting a table in a RDBMS. And you’d be correct. Gold star for you.

$sweet_collection = $db->name_of_collection;
$sweet_collection = $db->selectCollection('name_of_collection');

Sweet. Let’s do some tangible, hands-on exercises, mm’kay? Mm’kay.

The insert [note to self: penis joke here]

For this part, I’ll go with something stupid, like pretend that we’re creating a database of DVD’s. Cheezy, but it’ll do. First let’s set up some dummy data.

$dvd = array(
    'title' => 'A Nightmare On Elm Street',
    'slug' => 'anoes',
    'year' => '1983',
    'format' => 'Blu-Ray',
    'screen' => '16:10',
    'media' => array(
        'audio' => array(
            'English - DTS 5.2 HD',
            'Master Audio',
            'English - DD 1.0',
            'French - DD 1.0'
        ),
        'subtitles' => array(
            'english',
            'french',
            'spanish',
            'spanglish'
        ),
        'extras' => array(
            'Commentary Tracks',
            'Featurettes',
            'Alternate Endings',
            'Trivia Track'
        )
    ),
    'starring' => array(
        'Robert Englund',
        'John Saxon',
        'Heather Langenkamp',
        'Johnny Jepp'
    )
);

Yes, Nightmare on Elm Street. Johhny Depp’s debut movie. As you can see we have a pretty basic multidimensional associative array. The actual code to insert the data I want, you say with a Yoda type of grammar. Sure.

$sweet_collection->insert($dvd);

Boom. There is an optional second argument that can be used if you want to check if the insert succeeded or not. If set to true, it will return an array with the status of the insert. That is, the the array will be returned with the _id assigned by Mongo. Think about the lack of need to call insert_id() or similar. Sweet, huh? If the insert was not successful, it will return a boolean which represents if the array provided was not empty. Empty arrays will not be inserted. Let’s try that out. Assuming we just executed the insert above, that insert gets a unique id (the key is “_id” – try to echo out your last insert id with echo $dvd['_id'];) and then fire this little baby;

try {
    $sweet_collection->insert($dvd, true);
}
catch(MongoCursorException $e) {
    echo "Why you wanna play me like dat? That movie ID already exists in the DB!";
}

Pretty nice, huh? One more thing about the id – MongoDB creates a unique id if non is supplied manually. The MongoId is an object and not a string. It’s the MongoDB equivalent to an autoincrement in RDBMS. Each id is 12 bytes – 4+3+2+3. The first four bytes are a timestamp. The following three are a hash of the client machine’s hostname, the next two are “the lease significant bytes of the process id running the script” and the last three bytes are an incrementing value. Food for thought.

The update

As you may or may not have noticed, there are some minor mistakes in the data above. Oops. Guess we’ll have to update. Funny how these things work out! There are a couple of handy dandy modifier operations to make our Mongo life a little bit easier. The syntax is as follows;

update( [criteria], [objNew], [upsert], [multi] );

criteria – query which selects the record to update
objNew – updated object or $ operators (e.g., $inc) which manipulate the object
upsert – if this should be an “upsert”; that is, if the record does not exist, insert it
multi – if all documents matching criteria should be updated (the default is to only update the first document found)

Our criteria would, in this example, be a value we can search for. Let’s search for the slug, “anoes”. We need to update a couple of fields here; the year should be 1984, screen is 16:9, not 16:10, the english audio is 5.1 surround, not 5.2, obviously, and there is no ‘spanglish’ subtitle. Before we do the actual update, we should discuss the modifier operations that are available. Using these modifiers, we’re able to do atomic updates! Yes, that last sentence deserved an exclamation mark. Shut up.

$inc – increments the value by the factor you provide. If the value is not present, the key is set to that value.
$set – sets the value for a specified key.
$unset – deletes given key and value.
$push – append the specified value to the key if an array is present. If not, an array is set to the key. If the key is present but not an array, an error condition is raised.
$pushAll – append each value in the value array to the key. Works similar to $push.
$addToSet – adds value to the array only if it’s not in the array already.
$pop – removes the first or last element of the array. Use “1″ to remove the last, and “-1″ to remove the first.
$pull – removes all occurrences of the specified value from the key if the key is an array. If the key’s there but is not an array, an error is raised.
$pullAll – removes all occurrences of each value in the array given from the key. If the key is there but isn’t an array – you guessed it – an error is raised.

Still here? no?, I’ll wait some more then.

Okay, so how do we update our movie info? Like so;

$criteria = array(
    'slug' => 'anoes'
);

$objNew = array(
    '$inc' => array(
        'year' => 1
    ),
    '$set' => array(
        'screen' => '16:9',
        'media.audio.0' => 'English - DTS 5.2 HD'
    );
    '$unset' => array(
        'media.subtitles.3' => 1
    );
);

$sweet_collection->update(
    $criteria,
    $objNew
);

The last two arguments are not set since we’re not updating multiple records, nor do we want it to upsert. As you may have noticed, you can target a certain index in the array – see media.audio.0 and media.subtitles.3. Pretty nice. But what if you don’t know the index? Then you’re screwed. Go home.

No, not really. There is something called the positional operator and it’s represented by the $ sign. Use it to find an array entry and manipulate it. For example, let’s update the audio this way instead.

$sweet_collection->update(
    array(
        'media.audio.$' => 'English - DTS 5.2 HD'
    ),
    array(
        '$set' => 'English - DTS 5.1 HD'
    )
);

Note that the $ only applies to the first matched item in the query. This may be changed in the future, but as of now, no.

Fetch, Bobo, fetch! Good boy.

Okay, then. How do we actually fetch our data? In many ways, as it turns out. Queries in MongoDB support both conditional operators (less than, greater than, etc, no equals, in (much like SQL’s “in”), mod, and much more), regular expressions and has a terminology, or, rather, methods, that work similar to SQL syntax. Here’s a quick reference – the MongoDB method to the left and its MySQL equivalent to the right.

sort() - ORDER BY
limit() – LIMIT (chocking, I know)
group() – GROUP BY

Sorry, I thought that list would be longer. You also have the following to toy around with;

skip() – allows you to specify at which object the database should begin returning results. Cool. Think paging results.
count() – returns the number of objects. It counts at server level instead of client level.
snapshot() – this makes sure that no duplicates are returned or objects were missed which were present at both the start and at the end of the query’s execution – even if the objects were updated. Short query responses – less than 1 MB are aways snapshotted.

Having said that, now’s the time to get to know find() and findOne(). These functions lets you query a specified collection. Basically, you tell it which collection to query and which fields to return. Say we have a number of entries in our DVD database and we want to find titles released from 1980 up to 1990, we could write something like;

$years = array('x' => array( '$gte' => 1980, '$lt' => 1990 ));
$best_movies_evar = $sweet_collection->find( $years );

Conditional operators FTW. findOne() works the same way but it returns a single element instead. It returns the record or null if it fails. Let’s say you need to paginate your results as the last query return a massive amount of titles. And right now, you need ten results, starting from title number 30. No biggie.

$num_movies = $best_movies_evar->total();
$best_movies_evar->limit(10)->skip(30);
foreach ( $best_movies_evar $sweet_movie ) {
    var_dump( $sweet_movie );
}

Note that queries are actually not executed until the result is requested. That means you can refine your results before actually fetching them, like we did above. Got it? Nice. Niiccceeeeee.

Aww, that’s nice. C-click. Deleted!

Sooner or later, you’ll want to delete a record or two hundred. Easy enough.

$field_to_remove = array(
    'year' => 1984
);
$sweet_collection->remove( $field_to_remove );

This would remove any document with the year 1983. If you want to remove the first document found, pass true as the second parameter and you’re good to go, sparky.

Now you can create an index, too!

This should be filed under optimisation, bot whatever. You can create indexes in MongoDB which is just gravy. Let’s create an index based on year of release in our sweet sweet DVD collection database. It’s as simple as calling ensureIndex(). Pass an array with the keys you want o have as an index and either “1″ or “-1″ 1 for ascending and -1 descending.


$sweet_collection->ensureIndex(
    array(
        'year' => 1
    )
);

… and that’s it. There are however some options you may pass in an associative array in the second parameter.

unique – creates a unique index.
dropDups – If a duplicate value exists while creating an index, drop all but one value.
background – create indexes in the background while other operations are taking place. It’s synchronous by default, so set this to true to make it asynchronous.
safe – check if thecreation of an index succeeded. If it failed, a MongoCursosException is thrown
name – specify an index name, should Mongo complains about the index name being too long while you’re indexing many keys ot once.

So let’s say we want to create two indexes instead of just one, make it unique, have it work in th ebackground and see if it worked.

try {
    $sweet_collection->ensureIndex(
        array(
            'slug' => 1,
            'title => -1
        ),
        array(
            'unique' => 1,
            'background => 1,
            'safe' => 1
        )
    );
}
catch(MongoCursorException $e) {
    echo "Oh, hells noes!";
}

I think you get the general idea here. You can’t create unique indexes on non-unique values, of course. Shoudl we have two or more identical titles or slugs, the unique index will fail, whch actually is quite probable in this example. Fun, huh? That’ll teach you to copy paste code without checking what it does.

I think that’ll suffice for now. As a closing hint, I’d like to point you to a sweet package on phpclasses.org by Cesar D. Rodas. It’s called MongoFS and allows you to store and get data in MongoDB GridFS like files. See, MongoDB stores large objects as chunks of data as well as its metadata. It stores it in chunks are BSON objects in MongoDB are limited to 4 MB in size. The chunk approach allows you to fetch a range of data, which could come in handy. Of course, there are articles covering GridFS already, so I’ll just point you to them instead (by “them” I mean “this specific article” – deal with it). Definitely worth checking out.

No related posts.

A Popurls Clone with PHP, jQuery, Awesomeness

william — Tue, 23 Mar 2010 13:21:57 +0000

Good people of the Internets, I know a good deal of you frequently visit some sort of social website and/or aggregator such as Popurls, Reddit, Digg, Developer Zone, or what have you. After a deep and thorough analysis of my visitor statistics (read: pulling data out of my ass), many of you seem to like articles about PHP. This is not surprising, as PHP is a very popular (scripting) language. Then again, why anybody would want to read my incoherent ramblings is still a mystery to me. If you’re reading this, kudos! You officially have nothing better to do. Encouraging, isn’t it? Btw, Reddit, thanks for /r/nsfw/.

Since I have a crazy workload right now, I feel this is the perfect time to write a quick n’ dirty tutorial on how to build your very own Popurls. Impress your friends and/or boss with a nifty, hand made news aggregator. Yes, very buzz word friendly. This is part one of a series of articles where we start from this article’s code and later on expand it to employ a simple admin interface for the feeds as well as pulling our content and playing with it in Flash using ActionScript 3 and from there build a sweet little Flash widget. I am weirdly excited about this to the point of scaring myself, so let’s STFU and get started. If you want to see a demo, well, you’re in luck. Go there!

Abstract

What we’re going to do is use SimplePie to fetch an array of feeds of different formats, cache them and output the whole shebang. Then we’re gonna style it with some CSS and use some simple jQuery interactivity to sweeten the deal.

SimplePie is simple, not related to pi, tastes like it

We’ll start, obviously, with the back end. I should start with confessing that I’m an avid fan of the KISS and DRY principles. Instead of building our very own feed parser, we’ll use the excellent SimplePie class together with the IDNA Convert class. Don’t worry, the last one is bundled with the SimplePie 1.2 download. The IDNA Convert class helps us, amongst other things, to handle feeds where the domain name is internationalized according to RFC 3490, 3491, 3492 and 3454. IDNA is optional but as long as we already have it, why the fuck not?

First things first. Are you ready, willing and able to harness the awesomeness that is SimplePie? Assuming you have downloaded the package, unzipped/untarred it and uploaded to your web server/LAMP/WAMP, direct yourself to

http://[your_server]/[simplepie_path]/compatibility_test/sp_compatibility_test.php

Great success? Good, then let’s get started. As a side note, if you want to explore all the goodies that you get to play with , go to the SimplePie Documentation, which is a comprehensive API documentation.

Getting dirty

Before continuing, I should point out that SimplePie has automagical caching. It’s quite sweet. Just create a folder in the same directory as the SimplePie class called cache and chmod it so that the script may write to it. Let’s start by including the SimplePie library and the aforementioned IDNA Class as well ass initializing SimplePie and tell it which feeds to use. For the remainder of this article, I’ll assume that SimplePie is in the root of your folder and IDNA is in the folder idn.

require_once('simplepie.inc');
require_once('idn/idna_convert.class.php');

$popurls = new SimplePie();
$popurls->set_feed_url(
    array(
        'http://feeds.digg.com/digg/popular.rss',
        'http://www.reddit.com/.rss',
        'http://feeds.delicious.com/v2/rss/',
        'http://www.dzone.com/links/feed/frontpage/rss.xml',
        'http://feeds.wired.com/wired/index',
        'http://www.tuaw.com/rss.xml',
        'http://www.engadget.com/rss.xml'
    )
);

I twelve or so lines o’ code, we’ve done the dirty work for fetching the feeds that we want o play with. And who said dealing with code is hard? Pff. Thankfully, the SimplePie method set_feed_url accepts an array which makes our lives so much easier. Before getting down to the nitty gritty of displaying the feeds, you should start out with setting a parameter called set_item_limit. It does what you think, namely, limiting the number of items to return from each feed. If you don’t, chances are that you’ll have to wait for a long, long, looong time before actually seeing anything on your screen. That’ll make you come back here and add some snarky comment, claiming that the c0dez dun work lulz, n00b. Fair enough, the blog is called profeshunl newbie, but still. It’s about kharma. For our first proof-of-concept, a limit of three items per feed should suffice. While we’re at it, we’ll initialize the feeds and make sure that it’s served with UTF-8.

$popurls->set_item_limit(3);
$popurls->init();
$popurls->handle_content_type();

Not that hard, was it? Told you so. Now let’s set up a very simple XHTML structure below the PHP block in order to display the goods. For this part of the article, well settle for displaying the content in tables. Don’t try validating the code, though. It may seem a little bit slow the first time you runt this script, which is expected. When your browser’s done, check the cache directory. There should be seven .spc files there.




    p0wnurls
        


    p0wnurls

    

    get_items() as $feed_item ) {
        // Create a reference to the current item's parent feed
        $feed_parent = $feed_item->get_feed();
    ?>

        
            
                Favicon: get_favicon(); ?> (get_favicon(); ?>">)
            
            
                Permalink: get_permalink(); ?>
            
            
                Title: get_title(); ?>
            
            
                Date: get_date(); ?>
            
            
                Content: get_description(); ?>
            
            
                Description: get_description(); ?>
            
            
                Parent Permalink: get_permalink(); ?>
            
            
                Parent Title: get_title(); ?>

Still here? Nice. As you can see, more often than not the description is the same as the content. Also, we need to strip away the tags and crap that we don’t need. Basically, we only want the raw text. Let’s add a row to each table and see what we can come up with, mm’kay? Let’s try something like


    Description Stripped:get_description()), ENT_QUOTES, 'UTF-8'); ?>

Boom. Much better. html_entity_decode will convert html entities to their should-be characters. SInce we don’t need any images or, really, any inline elements, we’ll use strip_tags. Note that strip_tags does not in any way validate the HTML it parses. If the HTML is borked, it may lead to unexpected results, which is fancy talk for “it’ll blow up in your face and have you sleeping in the shower”. If you want to, you can always roll your own stripping mechanism with a regular expression. I wouldn’t recommend it, but if you’re the adventurous type or just don’t trust strip_tags, go ahead! The last link provides you with the following regex;

$regex = "/<\/?\w+((\s+(\w|\w[\w-]*\w)(\s*=\s*(?:\".*?\"|'.*?'|[^'\">\s]+))?)+\s*|\s*)\/?>/i";

…which should suffice as a starting point for you. And if you want to just strip whitespace, how about

$regex = '/(?:(?<=\>)|(?<=\/\>))(\s+)(?=\<\/?)/';

which was found over here.

Now that we know how to fetch the information we need, there is one thing that need to be sorted out. The default action for SimplePie is to aggregate all the feeds. This means that they’re all in one big list o’ items, sorted by date. Normally this would be fine, but we want a per feed sorting mechanism, right? Right. If you don’t you can safely skip this part. Then again, if you skip it, the rest of this article will be reeeally confusing.

Are we there yet?

I like playing with arrays, so let’s expand our initial code a bit. Let’s make an associative array of it and add a key for each feed. Let’s go with a nice name of the source domain. Manually for now. If you don’t like nested foreach loops, now’s your time to look away.

// Include SP
require_once('simplepie.inc');
require_once('idn/idna_convert.class.php');

// Instantiate a source array with domain short name as key and feed URL as value
$_source = array(
    'digg'=>'http://feeds.digg.com/digg/popular.rss',
    'reddit'=>'http://www.reddit.com/.rss',
    'delicious'=>'http://feeds.delicious.com/v2/rss/',
    'dzone'=>'http://www.dzone.com/links/feed/frontpage/rss.xml',
    'wired'=>'http://feeds.wired.com/wired/index',
    'tuaw'=>'http://www.tuaw.com/rss.xml',
    'engadget'=>'http://www.engadget.com/rss.xml'
);

// Instantiate empty array
$_sorted = array();

// Loop through the source array
foreach ( $_source as $_feed_key=>$_feed_value ) {

    // Instantiate new SP
    $_feed = new SimplePie();
    $_feed->set_feed_url($_feed_value);
    $_feed->init();

    // Loop through the items of the current feed
    foreach ( $_feed->get_items(0, 3) as $__item ) {

        // Add title, favicon and permalink to feed and items to the sorted array
        $_sorted[$_feed_key]['title'] = $_feed->get_title();
        $_sorted[$_feed_key]['favicon'] = $_feed->get_favicon();
        $_sorted[$_feed_key]['permalink'] = $_feed->get_permalink();
        $_sorted[$_feed_key]['feed_items'][] = array(
            'title' => $__item->get_title(),
            'date' => $__item->get_date(),
            'permalink' => $__item->get_permalink(),
            'content' => html_entity_decode(strip_tags($__item->get_description()), ENT_QUOTES, 'UTF-8')
        );
    } 

}

Short and sweet. Try a print_r( $_sorted ); and view source. Looks neat, doesn’t it? Note that set_item_limit has been removed in favor of specifying an offset and a limit to get_items() instead. set_item_limit wouldn’t work at all. As you can see from the date values, the items are sorted in ascending order. Jut the way we want it. The content part may be way too long though. We only need something like the first 100 characters of it to function as an excerpt. Let’s truncate the content. Yes, let’s. Add some sort of truncation function such as

// Truncate string
function truncate( $string, $length=100 )
{

    // Is it shorter than out string? If so, return it as is.
    if( strlen( $string ) <= $length )
    {
        return $string;
    }

    // No? Well, regex and substr the fuck out of that string and append an ellipsis at the end.
    return preg_replace('/\s+?(\S+)?$/', '', substr($string, 0, $length)).'...';
}

After that it’s only a matter of wrapping the value for the content key with the truncate function in the inner foreach loop

$_sorted[$_feed_key]['feed_items'][] = array(
    'title' => $__item->get_title(),
    'date' => $__item->get_date(),
    'permalink' => $__item->get_permalink(),
    'content' => truncate(html_entity_decode(strip_tags( $__item->get_content() ), ENT_QUOTES, 'UTF-8'))
);

Looking good, right? As you may notice, some feeds actually don’t have any content or just have a “submitted by…” kind of string, which no fun. Let’s add an ugly conditional that checks if the content is empty or, in this case, the feed is from Reddit. In that case, we’ll use the title as excerpt instead.

// Get the content
$_content = $__item->get_content();

// If it's empty or if it's Reddit...
if ( empty( $_content ) || $_feed_key == 'reddit')
{
    // Jut use the title
    $_content = $__item->get_title();
} 

$_sorted[$_feed_key]['feed_items'][] = array(
    'title' => $__item->get_title(),
    'date' => $__item->get_date(),
    'permalink' => $__item->get_permalink(),
    'content' => truncate(html_entity_decode(strip_tags( $_content ), ENT_QUOTES, 'UTF-8'))
);

Much better. However, we need to rearrange and restyle our code a bit. And lets get rid of those horrible tables and go with unordered lists instead.




    p0wnurls
    


    p0wnurls

    $feed )
    {
    ?>

        
        
            
                Favicon: (">)
            
            
                Permalink:
            
            
                     

                    
                        
                            Title:
                        
                        
                            Permalink:
                        
                        
                            Date:
                        
                        
                            Content:

So, are we there yet?

Starting to see the resemblance to Popurls? No? You douche. Well, actually, me neither. So lets fix that. First off, let’s add a container that we center on the screen. Then we’ll divide the feeds into columns of three. Then do some hacky crap to add a div after every third column. After that, we’ll divide each feed’s items in two – the top half is shown and the other part is hidden until we press a link. After that, we should style it like Popurls and include jQuery. Let’s begin with the body. This may seem a little messy. And that’s because it is.



    
    p0wnurls
    $feed )
    {
    ?>
        
            " alt="" />

Don’t go copy/pasting that code – as it turns out, my code highlighting script of choice in combination with WordPress strips away any plus signs, rendering the loops somewhat… lacking… Hm. Just check that $i and $j get incremented in the end of each loop and you should be fine. I added two very stealthy clues as to where to add the plus signs. Look for the clues, eagle eye, and report back later. Notice the rearrangement of code. We use the title for the permalink anchor as, well, the title, and put the content in a span. As a courtesy for all those of you who prefer to echo the content instead of multiple opening and closing php tags, here’s the inner loop for you to play with;

$j = 0;
foreach ( $feed['feed_items'] as $feed_item )
{
    if ( $j == $half_feed_items )
    {
        echo '
    
        
        
        ';
    }
    echo '
    
            ' . $feed_item['title'] . '
            ' . $feed_item['content'] . '
    
    ';
    $j  ;

    // Have we printed the last feed item? If so, close the ul up
    if ( $j == $total_feed_items )
    {
        echo '';
    }
}

Next up – CSS. This is nothing special and shouldn’t be hard to follow.

body {
        font-family: 'Lucida Grande', Helvetica, Arial, sans-serif;
        font-size:12px;
        text-align:center;
        background :#000000;
        color: #f3f3f3;
    }
    a {
        text-decoration:none;
        color: #83bacf;
    }
    #container {
        width:960px;
        margin: 0 auto;
        text-align:left;
    }
    .feed_container {
        float:left;
        width:320px;
        display:inline;
    }
    ul {
        padding: 5px 5px 5px 0;
        margin:0;
    }
    ul > li {
        list-style-type: none;
        margin:0;
        padding:0;
    }
    ul li span {
        display:none;
    }
    .feed_items li {
        padding: 5px 0;
        border-top: 1px solid #505050;
    }
    .more {
        display:block;
        background:#505050;
        height:10px;
    }
    .feeds_extra {
        display:none;
    }
    .clear {
        clear:both;
        padding: 0 0 10px 0;
    }
    #tooltip {
        position:absolute;
        border:2px solid #afa06b;
        background:#e4dd9c;
        padding: 5px;
        color:#333;
        display:none;
        width: 320px;
        text-align:left;
    }

Nothing fancy. The container has a width of 960 pixels and each feed gets a third each to play with. We added a dark background and colored the anchors blue and the tooltip yellowish. It matches Popurls color scheme. I think. We’ve added the necessary code for the tooltip as well. Speaking of tooltip, let’s include jQuery and the tooltip script. I’m going with jQuery 1.4.2 and a neat little tooltip that I found on css globe. The only problem is the way that script works. We have to alter it a bit in order to have it display our content in the tooltip. No biggie. We only need to change one line of code, namely line 21 from

this.t = this.title;

this.t = $(this).siblings().html();

so that it looks for content in our sibling span instead of the anchor’s title. Life’s good, right? Assuming that you put your javascript files in the root of your project, add some javascript in the head of the page

Simple. live() adds a handler to the event for all elements that match the selector no matte rif they’re in the DOM now or are added later. bind(‘click’, function(){ … }); would also do if that’s your preference. With me so far? I thought so. Let’s add the finishing touches to our main PHP code vlock so that it actually works. Who knew, right? We need to add two variables.

$total_feed_items = 10;
$half_feed_items = round( $total_feed_items/2 );

They’re used in the inner loop in the document body. Really, they are. We also need to change the hard coded value in the inner foreach loop to use our variable. Also note that we changed the number of feed items form 3 to 10.

foreach ( $_feed->get_items(0, $total_feed_items) as $__item )

To sum up, our PHP block should look something like this (comments removed)

function truncate( $string, $length=100 )
{
    if( strlen( $string ) <= $length )
    {
        return $string;
    }
    return preg_replace('/\s ?(\S )?$/', '', substr($string, 0, $length)).' ...';
}

require_once('simplepie.inc');
require_once('idn/idna_convert.class.php');

$_source = array(
    'digg'=>'http://feeds.digg.com/digg/popular.rss',
    'reddit'=>'http://www.reddit.com/.rss',
    'delicious'=>'http://feeds.delicious.com/v2/rss/',
    'dzone'=>'http://www.dzone.com/links/feed/frontpage/rss.xml',
    'wired'=>'http://feeds.wired.com/wired/index',
    'tuaw'=>'http://www.tuaw.com/rss.xml',
    'engadget'=>'http://www.engadget.com/rss.xml'
);

$_sorted = array();

$total_feed_items = 10;
$half_feed_items = round( $total_feed_items/2 );

foreach ( $_source as $_feed_key=>$_feed_value )
{
    $_feed = new SimplePie();
    $_feed->set_feed_url($_feed_value);
    $_feed->init();

    foreach ( $_feed->get_items(0, $total_feed_items) as $__item )
    {
        $_content = $__item->get_content();

        if ( empty( $_content ) || $_feed_key == 'reddit')
        {
            $_content = $__item->get_title();
        } 

        $_sorted[$_feed_key]['title'] = $_feed->get_title();
        $_sorted[$_feed_key]['favicon'] = $_feed->get_favicon();
        $_sorted[$_feed_key]['permalink'] = $_feed->get_permalink();
        $_sorted[$_feed_key]['feed_items'][] = array(
            'title' => truncate($__item->get_title()),
            'date' => $__item->get_date(),
            'permalink' => $__item->get_permalink(),
            'content' => truncate(html_entity_decode(strip_tags( $_content ), ENT_QUOTES, 'UTF-8'))
        );
    } 

}

YMMV. And the entire HTML page should look similar to this;




    
    p0wnurls

    
        
        
        



    
    p0wnurls
    $feed )
    {
    ?>
        
            " alt="" />

See, that wasn’t so hard, was it? The complete page is available here for your viewing pleasure. Again, note the lack of two plus signs in the inner loop. Instead of a nifty “More” button, we have a clunky grey bar. I leave it as an exercise for the reader to do something fun with it. That’s about it for this time. Next time we’ll see if we can’t make this code OO and add an admin interface to handle the feeds. Calm down, spanky, we’ll get there. Now I need to get some sleep. I’ve been watching Dexter a whole lot lately and he’s an inspiration, to say the least. The down side is that the show takes time from quality doing-jack-shit-on-the-Internet time. I have to see what I can do about that.

Hands-on tips for PHP security

Hands-on tips for PHP security

william — Thu, 11 Mar 2010 12:00:29 +0000

I got asked to review a fairly large piece of PHP code recently and, whoooo boy, was I in for a treat (treat as in clawing my eyes out with a rusty spoon while listening to Nickelback, as interpreted by Dr Zoidberg. In reverse). This magnificent piece of code was an eye opener in many ways and it made me feel a little bit better about myself, to be honest. It employed not only what would be called “bad practice”, but also a lot of plain ol’ stupidity. I know you are just dying for examples and I am not one to deny you the satisfaction. Also note that if this resembles your own code in any way, just leave because we don’t take kindly to you kinds of people, y’hear? And bring your cousin dad too. The beast contained gems such as

$connect_db = mysql_connect("localhost","username","password");
if (!$connect_db) {
    die('Could not connect becaus of ' . mysql_error());
} else {
    mysql_select_db("some_random_db", $connect_db);
    $command="INSERT INTO table_name ('username', /* some other crap */)
    VALUES ('".$_GET['user']."' /* more crappy crap crap */)";
    if (!mysql_query($command,$connect_db)){
        die('Could not add user because ' . mysql_error());
    }
    echo "Successfuly added user ".$_GET['user']."";
}

Spelling errors not omitted. No cup of heavenly blessed Kopi Luwak coffee, poured into a cup of golden banana leaves by naked triplets could counter the massive damage done to my corneas or, for that matter, my soul. Or what was left of it. The kicker was that the developer claimed it was OO and showed me the following “controller”. This is paraphrasing a bit, but you’ll get the general idea.

class MySite
{
    var $page;
    function MySite($page) {
        $this->page = $page;
        $content = "page.php?page=".$this->page;
        /*
        This is where page.php returned the entire document
        */
        echo $content;
    }
}
$init =  new MySite($_GET['page']);

Yum.

Needless to say, everything was passed around in the URL. Nothing was escaped as it “would hurt performance”. The e-mail function only checked that there was an @ in the string. You get the picture. Of course, a refactor, and I use the term loosely as it really was a steaming pile of vomit that would need a complete rewrite, was mission critical and the investors didn’t like the idea that the site would be down for maintenance for a while. As it turns out, this started as a hobby project that got the attention of somebody with money and decided that it would make a nice platform for a thriving business.

So this little article will deal with some security related matters such as input filtering. I expect there will be a second part to this as there is a plethora of things to consider, but this’ll do for now.

First off, I’d like to introduce you to a wonderful library without external dependencies called Inspekt. Mr. Ed Finkler of Funkatron built Inspekt upon the now deprecated Zend_Filter_Input, by Chris Shiflett.

[Insert Inspector Gadget joke here]

Inspekt acts as a firewall API between user input and the rest of the application. It takes PHP superglobal arrays, encapsulates their data in an “cage” object, and destroys the original superglobal. Data can then be retrieved from the input data object using a variety of accessor methods that apply filtering, or the data can be checked against validation methods.

Everybody wins! In order to take advantage of this black magic, all you need to do is something like this;

require "Inspekt.php";
$input = Inspekt::makeSuperCage();

Okay, what have we done here? When the library’s been required, we create an Inspekt_Supercage – that’s a single object that contains each input superglobal as an Inspekt_Cage. Inspekt_Cage objects encapsulate arrays of data. The original array is destroyed so you always have to access the data through the Input_Cage object’s methods. Say you need to check if $_POST['important_id'] really is an integer. Note that we are extending the introductory example above.

$input->post->testInt('important_id');

If it is not an integer, it will return false. More on that in a bit. What’s nice about Inspekt is that you can specify the kind of cage to create, i.e. a cage for just $_GET or $_POST, etc. You have the following tools to play with;

Inspekt::makeGetCage(); // Returns an Inspekt_Cage for the $_GET array
Inspekt::makePostCage(); // Returns an Inspekt_Cage for the $_POST array
Inspekt::makeCookieCage(); // Returns an Inspekt_Cage for the $_COOKIE array
Inspekt::makeServerCage(); // Returns an Inspekt_Cage for the $_SERVER array
Inspekt::makeFilesCage() // Returns an Inspekt_Cage for the $_FILES array
Inspekt::makeEnvCage() // Returns an Inspekt_Cage for the $_ENV array

Isn’t that nifty? The Inspekt_Supercage has no less than six public properties, such as get for $_GET, post for $_POST, etc. It correlates to the list just above. I promise. You access them by something like the following (again, extended from our first makeSuperCage example);

$input->get->testEmail('email');

If it’s valid, it will return the value. So, what else do we have in our toolbox? Well, the cage is all good and well, but we need some sort of properties to actually do the filtering or examining values. Let’s start with some tools for testing, shall we? The testers are neat methods to check the value on a given key. They return the value of the key if it’s good and false if it fails.

testAlnum (mixed $key)
// Returns value if every character is alphabetic or a digit

testAlpha (mixed $key)
// Returns value if every character is alphabetic

testBetween (mixed $key, mixed $min, mixed $max, [boolean $inc = TRUE])
// Returns value if it is greater than or equal to $min and less than or equal to $max.
// If $inc is set to FALSE, then the value must be strictly greater than $min and strictly less than $max.

testCcnum (mixed $key, [mixed $type = NULL])
// Returns value if it is a valid credit card number format.
// The optional second argument allows developers to indicate the type.

testDate (mixed $key)
// Returns $value if it is a valid date. The date is required to be in ISO 8601 format.

testDigits (mixed $key)
// Returns value if every character is a digit

testEmail (mixed $key)
// Returns value if it is a valid email format

testFloat (mixed $key)
// Returns value if it is a valid float value

testGreaterThan (mixed $key, [mixed $min = NULL])
// Returns value if it is greater than $min

testHex (mixed $key)
// Returns value if it is a valid hexadecimal format

testHostname (mixed $key, [integer $allow = ISPK_HOST_ALLOW_ALL])
// Returns value if it is a valid hostname

testInt (mixed $key)
// Returns value if it is a valid integer value

testIp (mixed $key)
// Returns value if it is a valid IP format

testLessThan (mixed $key, [mixed $max = NULL])
// Returns value if it is less than $max

testOneOf (mixed $key, [ $allowed = NULL])
// Returns value if it is one of $allowed

testPhone (mixed $key, [ $country = 'US'])
// Returns value if it is a valid phone number format. The optional second argument indicates the country.

testRegex (mixed $key, [mixed $pattern = NULL])
// Returns value if it matches $pattern. Uses preg_match() for the matching.

testUri (string $key)
// Returns value if it is a valid URI as defined in http://www.ietf.org/rfc/rfc2396.txt

testZip (mixed $key)
// Returns value if it is a valid US ZIP

Sexy, isn’t it? And in order to access your values after they’ve been filtered, you may use the following methods;

getAlnum (mixed $key)
// Returns only the alphabetic characters and digits in value.

getAlpha (mixed $key)
// Returns only the alphabetic characters in value.

getDigits (mixed $key)
// Returns only the digits in value. This differs from getInt().

getDir (mixed $key)
// Returns dirname(value).

getInt (mixed $key)
// Returns (int) value.

getPath (mixed $key)
// Returns realpath(value).

getPurifiedHTML (string $key)
// This returns the value of the given key passed through the HTMLPurifer object,
// if it is instantiated with Inspekt_Cage::loadHTMLPurifer

getROT13 (string $key)
// Returns ROT13-encoded version

getValue (string $key)
// Retrieves a value from the cage

getRaw (string $key)
// Returns value, unfiltered.

noPath (mixed $key)
// Returns basename(value).

noTags (mixed $key)
// Returns value with all tags removed.

noTagsOrSpecial (mixed $key)
// Returns value with tags stripped and the chars '"&<> and all ascii chars
// under 32 encoded as html entities

And of course, there is

keyExists (mixed $key)
// Checks if a key exists

It wouldn’t be complete if there where no methods for escaping strings prior to database calls. Well, aren’t we in luck?

escMySQL (mixed $value, [resource $conn = null])
// Escapes the value given with mysql_real_escape_string

escPgSQL (mixed $value, [resource $conn = null])
// Escapes the value given with pg_escape_string

escPgSQLBytea (mixed $value, [resource $conn = null])
// Escapes the value given with pg_escape_bytea

I should point out that you actually can wrap any array in Inspekt_Cage and not just superglobals, making the library a handy dandy tool for all your filtering and testing needs.

$dirty = array( /* imagine some random, unfiltered values here */ );
$johnny_cage = Inspekt_Cage::Factory($dirty);

to test some value in the array, just use

$johnny_cage->testInt('key');

To get even more nitty gritty, you can use static methods for those one off moments.

Inspekt::isAlnum (mixed $value)
Inspekt::isAlpha (mixed $value)
Inspekt::isBetween (mixed $value, mixed $min, mixed $max, [ $inc = TRUE])
Inspekt::isCcnum (mixed $value, [mixed $type = NULL])
Inspekt::isDate (mixed $value)
Inspekt::isDigits (mixed $value)
Inspekt::isEmail (string $value)
Inspekt::isFloat (string $value)
Inspekt::isGreaterThan (mixed $value, mixed $min)
Inspekt::isHex (mixed $value)
Inspekt::isHostname (mixed $value, [integer $allow = ISPK_HOST_ALLOW_ALL])
Inspekt::isInt (mixed $value)
Inspekt::isIp (mixed $value)
Inspekt::isLessThan (mixed $value, mixed $max)
Inspekt::isOneOf (mixed $value, [ $allowed = NULL])
Inspekt::isPhone (mixed $value, [ $country = 'US'])
Inspekt::isRegex (mixed $value, [mixed $pattern = NULL])
Inspekt::isUri (string $value, [integer $mode = ISPK_URI_ALLOW_COMMON])
Inspekt::isZip (mixed $value)
Inspekt::getAlnum (mixed $value)
Inspekt::getAlpha (mixed $value)
Inspekt::getDigits (mixed $value)
Inspekt::getDir (mixed $value)
Inspekt::getInt (mixed $value)
Inspekt::getPath (mixed $value)
Inspekt::noPath (mixed $value)
Inspekt::noTags (mixed $value)

Built in goodies

If you feel that using a library isn’t cool enough, or if I haven’t made the benefits clear, there are some things you ca do for filtering your values. I’m talking about the often overlooked PHP Filter functions. These are available if you are using at least PHP 5.2. Before continuing, I want to point you to the types of filters available. They’re validation filters, sanitizing filters, miscellaneous filters and, well, the predefined constants. First off is filter_var.

filter_var ( mixed $variable [, int $filter = FILTER_DEFAULT [, mixed $options ]] )

filter_var does what you think it does – it filters a variable with the specified filter. Go tautology! the $filter is specified in the list in the previous paragraph. The $options are an array of options or bitwise disjunction of flags. The function returns the filtered data or false if it fails. A quick example of e-mail validation could be

$is_it_okay = filter_var("some_dude\X00= _@ne@example..com", FILTER_VALIDATE_EMAIL);

Then there’s filter_var_array, which allows you to check an array of values.

filter_var_array ( array $data [, mixed $definition ] )

An example could be (borrowed from this place)

$data = array('bold', '', 'P*}i@893746%%%p*.i.*}}|.dw');
$myinputs = filter_var_array($data,FILTER_SANITIZE_STRING);
var_dump($myinputs);

//OUTPUT:
//formarray(3) { [0]=> string(4) "bold" [1]=> string(10) "javascript"
[2]=> string(26) "P*}i@893746%%%p*.i.*}}|.dw" }

Then we have filter_input which gets an external variable by name and optionally filters it.

filter_input ( int $type , string $variable_name [, int $filter = FILTER_DEFAULT [, mixed $options ]] )

$type may be either INPUT_GET, INPUT_POST, INPUT_COOKIE, INPUT_SERVER or INPUT_ENV ( INPUT_SESSION and INPUT_REQUEST are not implemented yet). Returns the requested variable on success, false if not or null if the $variable_name is not set (unless you use the flag FILTER_NULL_ON_FAILURE – in that case it returns false if the variable is not set and null if it fails). Say you want to sanitize $_GET['search'] for special characters. Try something like

$search = filter_input(INPUT_GET, 'search', FILTER_SANITIZE_SPECIAL_CHARS);

There is also filter_input_array, which does what you think it does – It filters an array instead of just a single variable.

$example_filters = array (
    "url" => FILTER_VALIDATE_URL,
    "ip" => FILTER_VALIDATE_IP,
    "temeraturep" => array(
        "filter"=>FILTER_VALIDATE_INT,
        "options"=>array(
            "min_range"=>1,
            "max_range"=>666
         )
    )
);

$sweetness = filter_input_array(INPUT_POST, $example_filters);

A quickie is filter_has_var, which checks if a variable of a specified type exists or not.

filter_has_var ( int $type , string $variable_name )

$type can be either INPUT_GET, INPUT_POST, INPUT_COOKIE, INPUT_SERVER or INPUT_ENV.

CRSFXSSFRSFCSFCSS X XSS SS XX C FFCCCE

A note on CRSF and XSS as well, while we’re at it. Inspekt allows you to run your data through the getPurifiedHTML method which basically runs it through HTMLPurifier and should provide a reasonable amount of XSS protection. If you have any tips on other XSS prevention projects or functions, please let me know. I stumbled upon a related project, csrf-magic. It uses PHP’s output buffering to “dynamically rewrite forms and scripts” and also check tokens in POST requests. You call it with

include_once 'csrf-magic.php';

It will even dynamically rewrite AJAX requests that uses XMLHttpRequest, if you want it to. Pretty sweet. It’s supposed to play along quite nicely with jQuery, MooTools, Prototype, Ext, Dojo, Yahoo UI Library, script.aculo.us and Prototype. There is also an option to set a configuration in the PHP script. Check out the following (borrowed form the bundled readme);

// This gets called if a csrf check fails
function my_csrf_callback() {
    echo "You're doing bad things young man!";
}

function csrf_startup() {

// While csrf-magic has a handy little heuristic for determining whether
// or not the content in the buffer is HTML or not, you should really
// give it a nudge and turn rewriting *off* when the content is
// not HTML. Implementation details will vary.
if (isset($_POST['ajax'])) csrf_conf('rewrite', false);

// This is a secret value that must be set in order to enable username
// and IP based checks. Don't show this to anyone. A secret id will
// automatically be generated for you if the directory csrf-magic.php
// is placed in is writable.
csrf_conf('secret', 'ABCDEFG123456');

// This enables JavaScript rewriting and will ensure your AJAX calls
// don't stop working.
csrf_conf('rewrite-js', '/csrf-magic.js');

// This makes csrf-magic call my_csrf_callback() before exiting when
// there is a bad csrf token. This lets me customize the error page.
csrf_conf('callback', 'my_csrf_callback');

// While this is enabled by default to boost backwards compatibility,
// for security purposes it should ideally be off. Some users can be
// NATted or have dialup addresses which rotate frequently. Cookies
// are much more reliable.
csrf_conf('allow-ip', false);

}

// Finally, include the library
include_once '/path/to/csrf-magic.php';

If you would like to roll your own CSRF solution, a simple one can be scripted on a coffee break. When dealing with forms, add a random token to your $_SESSION after calling session_start() such as;

$_SESSION['my_token'] = md5(uniqid(rand(), TRUE));

And then in the form itself, add a hidden input like so;

After submit, you can check if the token is valid with something simple as;

if ($_POST['my_token'] == $_SESSION['my_token']){
    /* Fun stuff */
}

You may ask, why don’t we limit the amount of time a token is valid? Yes, well, let’s do that then.

// Add token and time
$_SESSION['my_token'] = md5(uniqid(rand(), TRUE));
$_SESSION['my_token_time'] = time();

// Assume we now submitted the form, yadda yadda

$are_we_screwed = time() - $_SESSION['my_token_time'];

if ($_POST['my_token'] == $_SESSION['my_token']){
    /* It's all good! Or is it? */
    if ( $are_we_screwed < 60 ) {
        /* Less than a minute to play with */
    } else {
        /* Fail */
    }
}

There you have it. Some quick tips and/or things to think about. I’m not sure how I can end this little tidbit, so I’ll just paste a picture to keep you busy. Mm’kay?

Update: Reddit user pashj gave me a tip on adding a link to XSS examples so you can test you own project for, well, attack vectors. A classic cheat sheet full of examples is this one over at ha.ckers.org. Try it out! Let me know how it goes.

A Popurls Clone with PHP, jQuery, Awesomeness

Random thoughts on JavaScript. Part one of who knows. The basics.

william — Sun, 07 Mar 2010 15:42:48 +0000

This is going to be some sort of article on JavaScript. I’m still not sure exactly what will come of this, which means that now is your time to turn back. I won’t give you any more chances! Here’s something to keep you entertained while you think it through. And you may have to increase the volume while you watch it as it is just that mind numbingly awesome.

Still here? Cool. I wouldn’t be, but you know, to each his own.

People much smarter and more patient than me has written a lot about JavaScript. I guess that link should suffice as an article, but that would be a bit of a cop out on my part so I think I’ll give writing some random thoughts a try. It may just be a couple of pointers or tips. Or not. Hold my hand as we take the leap into the unknown.

First off, you know that you can try out your JavaScript online, right? Good. Now, let’s start with some JavaScript basics. And I mean really basic stuff.

Comment your code, but don’t go crazy with it.

var i = 0; // Declare i and set it to zero

A bag full of hurt. Use the good ol’ double slash for single line comments as per above. Use block comments for documentation or commenting out pieces of code.

/*
This is commented out
Isn't this pretty?
*/

Notation, notation, notation

In JavaScript, you can access properties of objects in two ways; square bracket notation and dot notation.

mySweetObject["property"] // Square bracket notation
mySweetObject.property // Dot notation

It may be tempting to use the dot notation as it’s a bit easier to type – because all coders are lazy, right? The downside of dot notation is that the property name is hard coded and thus can’t be altered at run time. Bracket notation, however, means that the property name is a string that gets evaluated. This means that you can use a variable, a hard coded string or a function call. Also, should the property be generated at run time, a bracket notation is required. Let’s start with a silly little example and say that you have a number of properties named “myValue1″, “myValue2″, “myValue3″ and so on. For some reason you need to loop through these properties using a for loop. If you try to access these properties like this

mySweetObject.myValue+i

you’ll fail. What you need is

mySweetObject["myValue"+i]

Boom, you’re in. So for the sake of consistency, use square bracket notation. Consistency is good. Repeat as a mantra.

Typecasting

The wonder that is JavaScript is weakly typed which means that you do not need to cast, or define, the type of data you are using. JavaScript, or rather the runtime system, performs automatic adjustments so you do not need to worry about it. This is either a good thing or a bad thing, depending on your background. Good for new programmers, bad for people used to C or Java (any language with strong typing). There are however times when it can be useful to typecast, such as when you are working with both strings and numbers. Since the plus sign is used both for string concatenation as well as regular addition, you might get some unexpected results. You can cast to three different types – Boolean, Number and String.

myBoolean1 = Boolean(666); // true
myBoolean2 = !! 666; // true
myString1 = String(666); // "666"
myString2 = "" + 666; // "666"
myNumber1 = Number("666"); // 666
myNumber2 = 1 * 666; // 666

You can try it out and make sure that it really gets typecasted by alerting with the help of typeof.

alert( typeof(myBoolean2) );

If we’ve behaved, that should alert “boolean”. The Boolean type returns true when the value passed to it is a string with at least one character, a number other than 0 or an object. E contrario it returns false when passed an empty string, the number 0, undefined or null. Number converts the entire value to a number, as compared to parseInt() and parseFloat(). The latter methods will only convert up to the first invalid character they encounter in a string. “6.6.6″ with parseInt() becomes “6″ and becomes “6.6″ with parseFloat(). Make a good habit of defining the radix with parseInt(). The second argument that you can pass to it and allows you to convert strings in binary, octal, hexadecimal or any other base into an integer. This may also be a good time to read up on floating point values. With Number() you get a “Not a Number” error, or the dreaded NaN. String should be a no-brainer. It’ll convert the value you want to a string, simple as that. What it does is call the toString() method. Note that String() will not result in an error for a null or undefined value – instead you get the string “null”.

Reserved words

This is something that can cause some confusion. I don’t know why, but I’ve seen my fair share of code where a developer uses a reserved word as a property name. It makes me kind of sad. The list of reserved words is a rather long one and pasting it here would just take up a lot of space, so instead I’ll just link to a page that has already done the dirty work. Please, please, please note the words that are reserved for future use, in the brand spanking new JavaScript 2.0 *cough* ECMA Script 4.0 *cough*. This will also leads us to…

Naming

See, there are limitations on the way can name things. You may use upper and lower case letters (a-z and A-Z), digits (0-9), and the underscore character (_). Although you can use other characters, it may break in certain cases. Better safe than sorry, right? Avoid starting names with the underscore character as it’s not rarely used to indicate privacy (more on that later on). Note that it doesn’t in itself create privacy.

Declare your variables

We know that JavaScript doesn’t require them to be declared, but, really, is it that hard? No. There is no excuse not to. You declare a variable with the use of the var keyword. Side note: If you come from C, you may need to know that JavaScript does not have block scope so be a good guy, and declare all your variables at the very top of your functions.

var myVar;
var myOtherVar = myRandomValue;

It makes it so much easier to identify undeclared variables that may, brace yourself, be implied globals. Implied globals can cause mayhem (mayhem as in a-minor-inconvenience-I-need-to-shape-up-my-code, not as in genocide). You see, the global object is the one namespace to rule them all. It holds, as you may have gathered by now, global variables and top level functions. It doesn’t take a mastermind to realize that collisions may occur and those collisions are part of the mayhem mentioned earlier. It will break your code. Basically, all JavaScript files you use run in the same scope which means that if you use globals, implied or not, and a script that loads after yours use the same variable or function names, the latter will replace your variables and functions. And that’s not good, is it? On a related note, you can declare multiple variables using a single var keyword and a comma separated list or variables.

var myVar1 = "Random",
    myVar2 = "Ramblings",
    myVar3 = "Fun stuff", myVar4, myVar5, myVar6;

Going global

Let’s back up for a moment and remember that in JavaScript, everything is an object. Or rather, a property of an object. In turn, an object is basically a collection of name-value pairs. The names are strings and values can be objects, numbers, booleans and strings. And a quick note on terminology – if the value is a function, strictly speaking, it’s called a method. But what about those functions, you ask? I’ve seen those around the code, you say. Well, those are actually properties to the window object.

function myFunction(myString) {
    alert(myString);
}

This little nifty function that alerts the variable string, actually means that we assigned a Function instance to the myFunction property. Functions do not have names. Actually, myFunction is a property of the window object. Does that make sense? It may be a bit more clear if we type it like so:

window.myFunction = new Function('myString', 'alert(myString)');

Feel a bit better now? Standalone functions are bound to the window object. When you would like to execute myFunction, you would normally type

myFunction('alert this string');

Nothing fancy. This can also be written as

window.myFunction('alert this string');

Remember, we are now dealing with global objects, which is in general a no-no, but nice for a simple explanation. Do you remember how we declared variables? When we type something like

var myVar = 'My string'

we are using a shorthand syntax (object literal – more on that a few lines down) for

var myVar = String('My string');

And these are also in the window object, i.e.

window.myVar = 'My string';
window.myOtherVar = String('My string');

Literal this, literal that

Thankfully, there are ways around the problem of collisions. One way is to employ object literals (see, I told you we’d get there) which basically means that you can define your own namespace (read: object or, going with OOP terms, a Constructor. Please note that it is Good Practice™ to capitalize the Constructor. I won’t in this case, because that’s just how i roll) and play with it. Or, as specified by the good people at Mozilla:

An object literal is a list of zero or more pairs of property names and associated values of an object, enclosed in curly braces ({})

Take this super simple example;

var myNameSpace = {
    myFunction:function(data) {
        alert(data);
    }
}

If you now try to call

myFunction('Alert this, you douche');

it will fail, as there is no property called myFunction in the window object. Instead, you need to call it using

myNameSpace.myFunction('Alert this, you douche');

That works just fine and if a script later on also has a myFunction() (preferrably in its own namespace), there will be no collisions. Goodie! The drawback of this technique is that you always have to go through the name of your object, or namespace. The solution would be to wrap your object in an anonymous function (or lambda). This will protect the scope. Our little snippet above would be typed like this:

var myNameSpace = function() {
    function myFunction(data) {
        alert(data);
    }
}();

Note the addition of the parenthesis on the last line and the ending semicolon. Also, you don’t need to stray away from the use of function myFunction() if that feels more comfortable. The parenthesis at the end makes the anonymous function execute right away. This is called the Module Pattern. This has the downside – or upside depending on your goals – of making myFunction() only available from within myNamespace. Any outside calls will fail (except maybe…). To fix this, you need to make a publicly available return statement.

var myNameSpace = function() {
    return {
        myFunction:function(data) {
            alert(data);
        }
    }
}();

Isn’t this fun? myNameSpace.myFunction(‘hey’); gets you going aaaand you’re pretty much back where we left off. Let’s alternate the example;

var myNameSpace = function() {
    function myFunction(data) {
        alert(data);
    }
    return {
        pointer:myFunction
    }
}();

See what we did there? We switched back to the other syntax for the private method and returned a pointer to that method instead. In this case, we call our function with

myNameSpace.pointer('hey');

Pretty sweet, right? The pointer can be as short as you want it and the private function can have any name you want, if that’s your preference. It makes me cringe a bit to mention public and private methods when dealing with JavaScript, but what the hell. Private methods or members are those that are created within the constructor. In the example above, myFunction is only accessible within the constructor myNameSpace and is returned only with the pointer or alias.

The reason that this works is due to the fact that JavaScript has closures. In essence, this means that a function defined within a function always has access to the vars and what have you of its outer function. Think about that for a while. That’s power, right there. Note that private and privileged methods can only be made when the object in questions is constructed. Public methods on the other hand, can be added ad hoc.

Friends with privileges

Although it might not be obvious, you also have the ability to create privileged methods. Privileged methods are accessible from outside the scope and can access private methods and variables. While you can delete or replace a privileged method, you can’t alter it. This is where you use the keyword this. This points to the method calling object.

function Constructor(data) {
    this.name = data;
}

This means that if we now construct a new object like so

var myConstructor = new Constructor('Gandhi');

myConstructor.name is “Gandhi”. Let’s expand.

function Constructor(data) {
    this.name = data;
    this.fetch = function() {
        return this.name;
    }
}

Okay, what can we do with this little gem of a function? Test it out with something basic like

var myConstructor = new Constructor('Gandhi');
alert( myConstructor.fetch() );

Did it work? Nice. It didn’t? Fail. Try again. Also try these quick examples:

var myConstructor1 = new Constructor('Gandhi');
var myConstructor2 = new Constructor('Shiva');
alert( "myConstructor1's name is "+myConstructor1.fetch() );
alert( "myConstructor2's name is "+myConstructor2.fetch() );

Did it work? Sweet. That’s about all I have to write for now. Fret not, there will be many more random thoughts coming quite soon, and not just on JavaScript. I’d like to leave you with an awesome talk from the JavaScript guru Douglas Crockford. Embedded below is a 45 minutes talk on Quality. It’s a good watch and, trust me, you’ll learn something new. It’ll make your code look better.

Douglas Crockford: “Quality” @ Yahoo! Video

No related posts.

CSS3 pseudo-class selector emulation for IE 5.5 – 8 and html5 with video

william — Mon, 01 Mar 2010 22:01:28 +0000

While gathering information as a part of my ongoing plan to be omniscient, if came across something that makes IE behave like a capable browser. Although to be fair, IE 8 is actually not that bad a browser – in the way that Star Wars Episodes 1-3 are not that bad. So, yeah, you do the math. I mean, Jar Jar. Pisses me off just thinking about it.

Mr Keith Clark has been conjuring up some serious magic and created a way to emulate CSS3 pseudo-class selectors alongside the javascript library of your choice. This means that you can use goodies such as :nth-child, :first-of-type, :first-child, :last-child, etc. It’s a simple as adding

… in your head tag. But how does it work, I hear you scream. Well, allow me to quote the man himself;

“First off, ie-css3.js checks the page for known JavaScript libraries and picks the one with the best CSS3 selector support.
Next, the style sheets on the page are downloaded and each of their selectors are parsed. Any CSS3 pseduo-classes are replaced by a standard CSS class of a similar name. For example: div:nth-child(2) will become div.iecss-nth-child-2. The detected JavaScript library is then used to find the DOM nodes matching the original CSS3 selector and the same CSS class is applied to them.
Finally, the original stylesheet is replaced with the new version and any elements targeted with CSS3 pseudo-classes will be styled.”

As of this writing, the :not selector isn’t supported but is on its way. There are some caveats, which, to be honest, are not that serious. Except that last bullet point. So don’t go all ajaxy on me now, y’hear?

You have to use a link tag (@import is OK)
page level style sheets will be ignored
Style sheets have to be on the same domain as the page itself and can not use file://
There is no dynamic emulation, which means that once the page has been rendered and the styles have been applied, changes in the DOM will not be updated.

While on the topic or, rather, something only slightly related, 52 framework promises a way to enable you to use the clean, semantic markup of html5 – today. It employs Eric Meyer’s CSS reset, which is nice in itself, and a grid system that is “loosely based on the 1kb grid and 960.gs”, and… Actually, why are you still here? Just get over there, and check out the 52 framework docs – with video as well as the demo. Don’t forget to check out the source code. can’t wait to play around with this in a project soon.

Come to think of it, I should mention a project on Google Code – HTML5 Video (or is it HTML5 Media?). This sweet litte package will enable you to embed MPEG-4 files using the H.264 video codec. Or, if you’re using Firefox 3.6, Ogv files encoded with Theora. If you use them both, you could sexy-up your code with the likings of

You just try to pretend that isn’t sexier than the horrible mess that is embed and object tags. Wrapping up, I might as well give you a tip about an awesome HTML 5 Canvas cheat sheet from the people at Nihilogic. You’re welcome.

CSS typography is fun. Really. If you’re in to S&M

william — Mon, 01 Mar 2010 13:58:02 +0000

So I found this article over at Smashing Magazine dealing with CSS typography. It’s a nice write up of both current and upcoming features – depending on your browser of choice – that will allow you to do some text formatting that I remember doing with Aldus PageMaker. Good times. Nice to see that technology is progressing at a steady pace. Competent writer Inayaili de León does a fine job going through some common and some not-so-common features such as hyphenation, indentation and hanging punctuation. I’m quite sure that you’ll find some new things in there.

She also covers the wonders of font-face, although I feel there is more to be written on that topic. Paul Irish, Esq, is a handsome fellow that makes jQuery even more fun. He has concocted a “bulletproof @font-face syntax” which works really, really well. Even in IE. Yes, in IE. Read all about it over here. Using these methods, you’ll want to use a font-face generator, such as this one @ fontsquirrel.com. It will make your online presence a bit more tolerable. But if you use Comic Sans, I will hunt you down. No, really. I’ll find a way.

Yeah, I tent do visit Smashing Magazine quite frequently even though I’m not on their payroll. Of course, they have written about typography before, and they have done so very well. One of my favorite articles is 50 Useful Design Tools For Beautiful Web Typography, covering everything from free fonts to which fonts are (usually) available on a given user’s computer. There’s just too much to mention here. Just go already! oh, and if you need some more CSS centric list o’ links, check this out instead.

If you’re into the fine art of controlling word wrapping (yeah, good luck with that), hyphenation and spaces, Alex Dunae has written an article that should make you feel all warm and fuzzy inside. What’s so special about his article, you ask? Well, my impatient, link loving visitor, he has actually made some screen shots with how different browsers interpret non-breaking space, soft hyphens and zero-width space. I know, crazy, right?

Speaking of CSS, CSS3 is shaping up quite nicely and it is a joy to play with. If you want some sort of example in the form of a sleek, sexy menu, you’re in luck! It also degrades quite nicely. As another example, you can actually create e sweet bokeh effect – using only CSS3 (and some jQuery). Also, here’s a cool effect that mimics a photo shoot, using jQuery and CSS. Should be quite nice for image cropping. And you really shouldn’t miss this article regarding multiple backgrounds and gradients with CSS! The important thing is to play and try out different techniques until you’re satisfied, look at it in IE 6, cry, ponder the meaning of life and then start over. That, my friend, is the circle of CSS.

Update: I forgot to mention a really sweet page with no less than 50 animations, using CSS3 (and some JS here and there). Definitely worth watching!

On a random note, here’s a newly hatched baby turtle. It resembles how I felt this morning.

Favicon:	get_favicon(); ?> (get_favicon(); ?>">)
Permalink:	get_permalink(); ?>
Title:	get_title(); ?>
Date:	get_date(); ?>
Content:	get_description(); ?>
Description:	get_description(); ?>
Parent Permalink:	get_permalink(); ?>
Parent Title:	get_title(); ?>