Friday 24 February 2012

cross-site xmlhttprequest with CORS

cross-site xmlhttprequest with CORS

 

CORS defines how browsers and servers communicate when accessing sources across origins using HTTP headers to allow both the browser and the server to know enough about each other to determine if the request or response should succeed or fail.





withCredentials

Standard CORS requests do not send or set any cookies by default. In order to include cookies as part of the request, you need to set the XmlHttpRequest’s .withCredentials property to true:
xhr.withCredentials = true;
In order for this to work, the server must also enable credentials by setting the Access-Control-Allow-Credentials response header to “true”. See the server section for details.
Access-Control-Allow-Credentials: true
 
 


http://hacks.mozilla.org/2009/07/cross-site-xmlhttprequest-with-cors/
http://www.html5rocks.com/en/tutorials/cors/
http://enable-cors.org/


======================
ADD HEADERS

For Apache

Apache can be configured to expose this header using mod_headers, this is enabled by default in Apache however you may want to ensure it's enabled by running the following command:

a2enmod headers

To expose the header you simply add the following line inside , , or sections, or within a .htaccess file:

Header set Access-Control-Allow-Origin *

* = ALL ...you can specify a domain only

Note: you can also use add rather than set, but be aware that add can add the header multiple times, so it's likely safer to use set. Eventually, you may need to reload Apache to make sure your changes are applied.



======================




// Create the XHR object.
function createCORSRequest(method, url) {
  var xhr = new XMLHttpRequest();
  if ("withCredentials" in xhr) {
    // XHR for Chrome/Safari/Firefox.
    xhr.open(method, url, true);
  } else if (typeof XDomainRequest != "undefined") {
    // XDomainRequest for IE.
    xhr = new XDomainRequest();
    xhr.open(method, url);
  } else {
    // CORS not supported.
    xhr = null;
  }
  return xhr;
}

// Helper method to parse the title tag from the response.
function getTitle(text) {
  return text.match('(.*)?')[1];
}

// Make the actual CORS request.
function makeCorsRequest() {
  // bibliographica.org supports CORS.
  var url = 'http://bibliographica.org/';

  var xhr = createCORSRequest('GET', url);
  if (!xhr) {
    alert('CORS not supported');
    return;
  }

  // Response handlers.
  xhr.onload = function() {
    var text = xhr.responseText;
    var title = getTitle(text);
    alert('Response from CORS request to ' + url + ': ' + title);
  };

  xhr.onerror = function() {
    alert('Woops, there was an error making the request.');
  };

  xhr.send();
}

XMLHttpRequest

var client = new XMLHttpRequest();
client.open("GET", "http://example.com/hello")
client.onreadystatechange = function() { /* do something */ }
client.send()

Modernizr Test Suite

http://modernizr.github.com/Modernizr/test/

HTML5 Cross Browser Polyfills

So here we're collecting all the shims, fallbacks, and polyfills in order to implant html5 functionality in browsers that don't natively support them.
The general idea is that: we, as developers, should be able to develop with the HTML5 apis, and scripts can create the methods and objects that should exist. Developing in this future-proof way means as users upgrade, your code doesn't have to change but users will move to the better, native experience cleanly.
Looking for a way to conditionally load these scripts client-side based on feature detects? See Modernizr. Looking for a guide to writing your own polyfills? See Writing Cross-Browser JavaScript Polyfills.

The All-In-One Entirely-Not-Alphabetical No-Bullshit Guide to HTML5 Fallbacks

SVG

  • svgweb by brad neuberg & others
    • fallback via Flash
  • Raphaël by dmitry baranovsky
    • abstracted API. adds features. fallback for IE via VML.
  • Ample SDK by Sergey ilinsky
  • canvg by Gabe Lerner
    • writes SVG to canvas. Good for Android.
  • SVG Boilerplate alpha and still buggy, but handle multiple concurrent SVG shims together
  • SIE SVG library by dhrname
    • fallback to VML for oldIE
  • dojo gfx by Eugene Lazutkin, Kun Xi, Chris Mitchell
    • fallback via VML, Canvas, Silverlight and Flash
  • fabric.js by kangax
    • can render SVG via canvas.
  • inline SVG polyfill by mstalfoort

SVG SMIL

Canvas

Web Storage (LocalStorage and SessionStorage)

Non HTML5 API Solutions

Sectioning Elements

  • html5shiv by jon neal, afarkas & community
    • enables for print use, as well.
  • innerShiv by joe bartlett
    • enables elements for innerhtml use

Video

  • html5media
    • Uses flowplayer swf for fallback.
  • mediaelement.js by John Dyer
    • swf player that mimics HTML5 media API. Consistent UI across browsers and flash. <track> support.
  • SublimeVideo
    • World's first zero maintenance HTML5 video player as a service solution.
  • video.js by Zencoder
    • Device fixes. <track> support
  • video for everybody by Kroc Camen (orig.)
    • The granddaddy of <video> fallbacks. Markup-based solution, works without javascript. A must read.

VTT: Video Timed Track (subtitles)

  • videosub
  • JSCaptions by Daniel Davis
  • Captionator by Christopher Giffard (Supports WHATWG TextTrack spec & has experimental support for MediaTracks/synchronised media elements)
  • playr by Julien Villetorte (support both WebVTT & SubRip files)

Audio

  • jPlayer custom player with jQuery dependency, flash fallback
  • audio.js custom player, flash fallback by ab+c
  • SoundManager2 custom player, flash fallback, api, extensive documentation

Web SQL Database

Local Database Abstractions (non-HTML5 APIs, but useful)

Web Forms

Web Forms : input placeholder

Accessibility / ARIA

Web Workers

Web Sockets

Geo-Location

Application Cache

Browser State Management

Offline & Online Events

  • offline-event.js by Remy Sharp (Opera not supported, and Safari doesn't update navigator.onLine property)

HTML5 History API (pushState, replaceState, popState)

  • History.js by Benjamin Lupton - (optional hashchange fallback for HTML4 browsers)
  • * PJAX (pushState + ajax = pjax) by Chris Wanstrath (not a polyfill)

Hashchange Event

EventSource

Animated PNG (APNG)

  • APNG-Canvas by Давид Мзареулян
  • png.js by Devon Govett
    • Parses and renders PNGs both animated and static to a canvas element with putImageData

Ruby

<details> and <summary>

<meter>

Support for <output>, <progress>, <menu>, <command>, <keygen> elements..

dataset property (for use with custom data-* attributes)

<datalist>

<ol reversed>

<style scoped>

  • Scoped Styles by Simon Madine
    • jQuery Plugin to enable the scoped attribute on style blocks so they only affect their parent element's children.

File API / Drag and Drop

Base64 (window.atob and window.btoa)

Please also note that strings out of the ASCII range need an additional layer of encoding in JavaScript.

Device Access via <device>

WebGL

MathML

(Web) Audio (Data) API

classList

Cross-Document/Domain Messaging (postMessage)

  • easyXDM by Sean Kinsey
    • Implements XDM and RPC in most browsers since IE6. Also ships with an endpoint for exposing ajax across the domain boundary.
  • postMessage jquery plugin by ben alman.

Cross-Origin Resource Sharing (CORS)

  • pmxdr by Eli Grey.
    • Implements a cross-domain XHR. Note that sites being requested must have a pmxdr host.
  • postmessage-proxied-xhr (PPX) by Atul Varma.
  • flXHR by kyle simpson
    • Provides XHR API using Flash transport mechanism. Target server must have a crossdomain.xml policy file.

DOM

DOM Range and Selection

DOM Parsing and Serialization

ECMAScript 5

  • es5-shim by kris kowal, tom robinson, community.
  • Xccessors by Eli Grey.
    • Implements ES5 accessors (getters and setters) for browsers supporting legacy methods and/or legacy methods for browsers supporting ES5.
  • getters and setters by the IE team.
    • pick up get/set'ers for DOM elements in IE8
  • json2.js by doug crockford
  • ddr-ecma5 by David de Rosier
  • augment.js by Oliver Nightingale

ECMAScript 6 (Harmony)

CSS3 Selectors

CSS3 Transforms

CSS3 Styles

  • cssFx - by Ivan Malopinsky
    • Standalone, lightweight, client-side JS; supports almost all CSS3 properties
  • CSS Crush
    • A PHP based CSS pre-processor for handling vendor prefixes, CSS variables, data-uris and minification
  • CSS Agent by Keith Clark
    • A ASP.Net based CSS pre-processor for handling vendor prefixes, CSS variables and minification
  • jQuery Css3 Finalize by Han Lin Yap
  • CSS3 PIE by Jason Johnston
    • CSS3 decoration rendering for IE 6-9. Supports: border-radius, box-shadow, multiple backgrounds, linear gradients, border-image
  • Flexie by Richard Herrera
    • Flexible box model. (flexbox) Supports: box-orient, box-align, box-direction, box-pack, box-flex, box-ordinal-group
  • CSS Template Layout jQuery plug-in by Alexis Deveria
  • CSS3 - Multi Column by Cédric Savarese
    • supports: column-count, column-width, column-gap, column-rule
  • jQuery Text Shadow by heygrady
    • text-shadow action via jQuery plugin API
  • jQuery Text Overflow plugin by Devon Govett, Andrew A. Kononykhin, Micky Hulse
  • Regions.js by Robin Ricard
  • borderBoxModel by Alberto Gasparin
    • box-sizing: border-box; javascript polyfill for IE6+
  • box-sizing-polyfill by Christian "Schepp" Schaefer
    • box-sizing: border-box; for IE6 and IE7 via Microsoft's proprietary CSS behavior / .htc-script
  • Borderbox for Compass by Richard Herrera
    • box-sizing: border-box; polyfill for SASS/Compass. Outputs equivalent CSS for IE6 & 7
  • Units by Grady Kuhnline
    • Units is a JavaScript library for converting between angle and length units (including the 'rem' unit). Both are intended for use within other libraries that need to convert between acceptable CSS units when creating polyfills.

CSS3 Media Queries

  • css3-mediaqueries-js
  • Respond by Scott Jehl
    • A fast & lightweight polyfill for min/max-width CSS3 Media Queries (for IE 6-8, and more)
  • mediatizr by Aurélien Delogu
    • Responsive-oriented CSS3 media queries polyfill
  • matchMedia by Scott Jehl, Paul Irish, Nicholas Zakas
    • A standardized way to check media query matching in script.
  • jQuery Media Helpers by Scott Jehl
    • Some media query and cross-browser responsive design helpers, abstracted from jQuery Mobile

CSS4 Selectors

Microdata API

  • microdatajs by Philip Jägenstedt
  • Microdata-JS pure JS (Extending Element.prototype with IE6+ support) by Егор Халимоненко

XBL

<link rel="prefetch|prerender" ...

  • jiagra by samy kamkar
    • a psuedo-polyfill for preloading pages to browser cache

Flash

  • Google Swiffy - webapp converts SWF to HTML+JS. Supports subset of SWF 8 and ActionScript 2.0, and webkit-only output.
  • Adobe Wallaby desktop app converts FLA to HTML+JS.
  • Gordon javascript that interprets SWFs live. Plays only SWF1 files completely, as well as most of the SWF2 features.

Visibility

  • visibility.js by Evil Martians
  • visibly.js by Addy Osmani
  • Page Visibility shim for jQuery by Mathias Bynens
    • The HTML5 Page Visibility API allows you to determine whether your web page is visible to the user, is hidden in background tab or is prerendering. visibility.js deals with vendor-prefixes and provides a fallback for browsers without this API by assuming the page is always visible. However, this is a wrapper more than a polyfill.

Hyphenation

  • hyphenator by Mathias Nater
    • CSS3 defines six properties and an at-rule that influence hyphenation. hyphenator is a JavaScript implementation with pattern files for many languages.

Media Capture

Other

Mixed!

  • Webshims Lib several Polyfills using lightweight capability based loading concept (shiv, innershiv, canvas, forms, ES5, localStorage/sessionStorage, JSON, geolocation) by Alexander Farkas
  • Assorted file formats decoding of several file formats done in javascript, documented on the excellent jswiki
  • Polyfill.js Polyfills loaded dynamically based on need (list of available polyfills on the linked page) by James Brumond
  • ES5/DOM shim implements DOM API and EcmaScript 5 (and some from ES6), and bug fixing in IE6+ and other browsers
Everything!
  • google chrome frame
    • not exactly a feature shim, but.. well.. it'll help. :)
  • ie7.js, ie8.js, ie9.js by Dean Edwards
    • A JavaScript library to make MSIE behave like a modern browser. Many fixes including PNG transparency, CSS styles/selectors, rendering bug fixes, etc.

Contributors

Edited and maintained by Paul Irish. Contributions from Jonathan Cook, Mark Boas, Michael Behan, Mathias Bynens, Eli Grey, Øyvind Sean Kinsey, and others.


https://github.com/Modernizr/Modernizr/wiki/HTML5-Cross-Browser-Polyfills#wiki-CORS

Tuesday 21 February 2012

Profiling with XHProf

If there's something we developers are really bad at, it's guessing. We think we know which parts of our application are slow, and spend a lot of time optimising those, but in reality the bottlenecks are often somewhere else. The only sane thing to do is measuring, with the help of some profiling tools.
There are a few profilers available for PHP, the most commonly used being Xdebug, which combined with KCacheGrind/WinCacheGrind/MacCallGrind can show the function call graph and the time spent in each function.
In this article, we're going to try another profiler, XHProf, developed at Facebook and open sourced in March 2009 (under the Apache 2.0 license). XHProf is a function-level hierarchical profiler, with a PHP extension (written in C) to collect the raw data, and a few PHP scripts for the reporting/UI layer.

According to Wikipedia:
profiling, a form of dynamic program analysis (as opposed to static code analysis), is the investigation of a program's behavior using information gathered as the program executes. The usual purpose of this analysis is to determine which sections of a program to optimize - to increase its overall speed, decrease its memory requirement or sometimes both
So a profiler is a tool that records the program events as they happen, and their effect on the system, collecting data with many different techniques. Some profilers only measure memory and CPU utilisation, others gather a lot more information, like full function call traces, times, and aggregate data. They can be flat or hierarchical, i.e. they can analyse each function by itself or in its context, with the full tree of its descendents.

Installation

At the moment, XHProf is only available for Linux and FreeBSD (and is expected to work with Mac OS).
The easiest way to get it is via the PEAR installer (package home):
apt-get install php5-common
 
pecl config-set preferred_state beta
pecl install xhprof
If it complains because it can't find config.m4, you can still build the extension manually, using the following steps:
wget http://pecl.php.net/get/xhprof-0.9.2.tgz
tar xvf xhprof-0.9.2.tgz
cd ./xhprof-0.9.2/extension/
phpize
./configure --with-php-config=/usr/local/bin/php-config
make
make install
make test
Once you have XHProf installed, you should enable it. Open your php.ini and add
[xhprof]
extension=xhprof.so
xhprof.output_dir="/var/tmp/xhprof"
Where /var/tmp/xhprof is the directory that will collect the profile data for each run.
Restart apache, and the XHProf extension should be enabled (check with "php -m" that this is the case).

Profile a Block of Code

To profile a block of code, wrap it around these two calls:
// your code
 
// start profiling
xhprof_enable(XHPROF_FLAGS_CPU + XHPROF_FLAGS_MEMORY);
 
// the code you want to profile
 
// stop profiler
$xhprof_data = xhprof_disable();
It is possible to dump the $xhprof_data array to view the raw profiler data for each function call (number of calls, wall time, CPU time, memory usage, peak memory usage) if you want to inspect these at any point.
xhprof_enable() accepts some flags to control what to profile: by default only call counts and elapsed time are profiled, you can add memory and CPU utilisation; make sure they're enabled in your dev environment (but disable the CPU timer if you use it in production, as it adds a high overhead). If you find the output too noisy, you can disable the reporting of builtin PHP functions with the XHPROF_FLAGS_NO_BUILTINS flag, or even exclude specific functions, by passing a second parameter like this:
// ignore builtin functions and call_user_func* during profiling
$ignore = array('call_user_func', 'call_user_func_array');
xhprof_enable(0, array('ignored_functions' =>  $ignore));

Profile an Entire Page

It's usually more useful to have a complete overview of the page, rather than a small block of code, and it's probably better to have such an overview formatted as a table or a graph, as opposed to an array dump. For this purpose, XHProf provides a convenient UI that must be enabled in order to be used.
The code for the XHProf UI can be found in the xhprof_html/ and xhprof_lib/ directories. Assuming they are created in /usr/local/lib/php/ , we can symlink that directory to /var/www/xhprof/ so it's available from our DocumentRoot.
We also need to create two PHP files:
/usr/share/php5/utilities/xhprof/header.php
<?php
if (extension_loaded('xhprof')) {
    include_once '/usr/local/lib/php/xhprof_lib/utils/xhprof_lib.php';
    include_once '/usr/local/lib/php/xhprof_lib/utils/xhprof_runs.php';
    xhprof_enable(XHPROF_FLAGS_CPU + XHPROF_FLAGS_MEMORY);
}
/usr/share/php5/utilities/xhprof/footer.php
if (extension_loaded('xhprof')) {
    $profiler_namespace = 'myapp';  // namespace for your application
    $xhprof_data = xhprof_disable();
    $xhprof_runs = new XHProfRuns_Default();
    $run_id = $xhprof_runs->save_run($xhprof_data, $profiler_namespace);
 
    // url to the XHProf UI libraries (change the host name and path)
    $profiler_url = sprintf('http://myhost.com/xhprof/xhprof_html/index.php?run=%s&source=%s', $run_id, $profiler_namespace);
    echo '<a href="'. $profiler_url .'" target="_blank">Profiler output</a>';
}
Finally, we add them to an .htaccess file so they're automatically prepended/appended to our pages:
php_value auto_prepend_file /var/www/xhprof/header.php
php_value auto_append_file /var/www/xhprof/footer.php
At the bottom of your pages, you should now have a link to the profiler output. This is a huge time saver, because every time you load the page, you have fresh profiler data, only one click away, and it doesn't require having external tools to parse and analyse it.

How to Use XHProf UI

If you click on the link at the bottom of the page, a new page opens with the profiler data:
XHProf UI
As you can see, the page has a nice summary with overall statistics, and a table with all the function calls, that can be sorted by many parameters:
  • Number of Calls
  • Memory Usage
  • Peak Memory Usage
  • CPU time (i.e. CPU time in both kernel and user space)
  • Wall time (i.e. elapsed time: if you perform a network call, that's the CPU time to call the service and parse the response, plus the time spent waiting for the response itself and other resources)
Memory usage and CPU time are further differentiated into "Inclusive" and "Exclusive": Inclusive Time includes the time spent in the function itself and in all the descendant functions; Exclusive Time only measures time spent in the function itself, without including the descendant calls.
Finally, in the report page there's also an input box to filter by function name, and a link to the full call graph, similar to the one you would get with *CacheGrind. Make sure you have GraphViz installed (apt-get install graphviz).
Callgraph
As stated in the documentation, XHProf keeps track of only one level of calling context and is therefore only able to answer questions about a function looking either one level up or down. This is rarely a problem, since you can drill down or up at any level. Clicking on a function name in fact will show details about the current function, the parent (caller) and the children (called).
Drill down view
As a rule of thumb, when we're ready to optimise our application, we should start sorting the data by CPU (exclusive), and look at the top of the list. The functions at the top are the most expensive ones, so that's where we should focus our efforts and start optimising/refactoring. If there's nothing obviously wrong, we drill-down and see if there's something more evident at an inner level. After every change, we run the profiler again, to see the progress (or lack thereof). Once we are happy, we sort by Memory Usage or Wall time, and start again.
Here's a quick summary if you want to print a step-by-step worksheet as a reference:
  • Profile;
  • Sort by CPU/Memory usage, time (exclusive) and function calls;
  • Start from the top of the list;
  • Analyse, refactor and/or optimise;
  • Measure improvement;
  • Start over. Again, and again, and again.
Profiling can be an extremely tedious process, because it requires a lot of patience, and a lot of time staring at numbers in a table (how exciting, eh?). Hopefully, the results of this process are exciting: improvements are often dramatic, since rewriting the slowest parts of the code (and not those we think are slow) has a considerable effect on the overall page load and ultimately on the user's experience. The advantage of using good tools is that they help maintaining discipline and focus, and thus in building experience.

Diffs and Aggregate Reports

XHProf has a nice feature to get the differences between two different runs, clearly marked in red and green colours. This way it is easy to instantly see the improvements after every change.
To view the report use a URL of the form:
http://%xhprof-ui-address%/index.php?run1=XXX&amp;run2=YYY&amp;source=myapp
Where XXX and YYY are run ids, and the namespace is "myapp" .
Diff between runs
Also, it's possible to aggregate the results of different runs, to "normalise" the reports. To do so, separate the run IDs with a comma in the URL:
http://%xhprof-ui-address%/index.php?run=XXX,YYY,ZZZ&amp;source=myapp

The Path to Scalability

Measure the Baseline

As in every journey, you must know where you are and where you want to go. If you need your application to scale, you must know what your targets are (users/sec, memory usage, page generation time), as well as your constraints (of your application, framework, server resources).
Before even starting coding your application, it's a good idea to measure the baseline of your framework, if you use one. For instance, here's a summary of an empty Zend Framework project (NB: the same considerations apply to any framework, I do NOT intend to single out ZF as a bad framework):
Profile for ZF
This tells you that unless you optimise the framework itself or cache the full page, you can't use less than 2.5MB of memory or have less than 1500 function calls per page load. This is your starting point.
Profiling the framework itself is not just an exercise in style, but is an eye opener on how it works and how (in)efficient the various components are, so any time you decide to use one, you know what to expect.
There are many examples of common programming practice where you might be surprised to see the impact these choices have; here are some examples.
If you have a config.ini setting called "error.logging.level", and use Zend_Config to read its value, you need to use $config->error->logging->level. Every "arrow" operator means two function calls. So that's 6 function calls just to read the value of a config setting. If you read that value often or in a loop, consider saving it into a variable.
Every time you use a view helper, there's a lot of stuff going on behind the scenes; here's the call stack (it's actually much worse, but you get the idea):
View helper call stack
When you call partial() to render a template, the current view object is cloned, and all the non-private variables are unset. This is done through expensive reflection and an awful lot of substr() calls. Use render() instead if you can (or a view helper if the template is really small and called many times).
Every time you render a template or use a model class, ZF scans the include path to find the correct file to load, even if you already requested that file before. You'll be surprised to know how many stat calls are made in a single page execution: thousands! Luckily, with XHProf (or even with strace/dtrace, in this case) it's easy to see whenever a file is read from disk, so you can optimise the include_path order, and possibly use APC to avoid scanning the include_path twice for the same file.
Every time you use Zend_Json::encode() instead of json_encode(), unless you have a very specific reason to do so, you should hit yourself with a stick (perhaps not literally). Profiling the call and seeing what happens is left as an exercise to the reader.
As I said, I don't intend to bash Zend Framework, I'm sure the others are no worse/better. What is important though is to be aware of the cost of each component of your framework, so you can make a conscious decision on which building blocks to use in your application.

Identify Bottlenecks

It is likely that your application will access external resources: a database, a web service, or data on disk. These are usually the most expensive operations you should try to minimise. If you don't see them at the top of the list when you look at the XHProf reports, it probably means that there's something wrong: in this case the framework might be the main bottleneck, or you need to refactor your architecture.
Sometimes, there's no single call eating all the resources, but it's easy to spot a cluster of function calls related to a certain part of the code:
Clusters of entries
Needless to say, this is a clear indicator that you must refactor that component.

Do Less. Do Nothing. Reuse.

When you identify a slow piece of code, before optimising it, rethink why you are doing something, if it's the right place to do it, and if possible reduce the amount of data you need to process. Only after these steps you can start worrying about the best way to do it.
I'm sure we all agree on the above statement, but sometimes it's not that obvious what to look for. Or we think we already optimised everything, the reports don't show any single resource hog, and we reached a dead end. This is when I find it useful to sort the XHProf reports by number of function calls. Usually, it is not a good indicator of the performance of a piece of code, because a single function responsible to retrieve data from an external source is a few orders of magnitude slower than many calls to an internal PHP function, for instance. On the other hand, even if PHP is fast, do we really need to call strtolower() 15000 times? Looking for odd things like this might give some hints on how we process data, and maybe come up with a better way. Too often we tend to bash a language for its slowness, and we tend to forget that usually performance issues have more to do with the implemented algorithms than with the operations used.
Here are some other code smells that might suggest we are doing something in a sub-optimal way:
  • Immutable functions called within loops
  • Same content being generated twice
  • Content that don't change being generated every time
  • Content being generated even if not used
All these cases are perfect candidates for caching. Of course I'm not suggesting caching everything. Remember that memory is another limited resource, so don't abuse it if you need to scale; the key is to spread the load uniformly across all the available resources. You have to think about the cache-hit ratio, and start caching things you hit all the times. Also, it makes little sense to cache if it takes more effort writing to the cache than you save. But more often than not, you can cache a LOT of content.
In order of effectiveness, you can use static variables, APC, memcached. But do not forget about other kind of caches that are even more effective: proxy cache (or reverse-proxy), and of course the user's browser. If you send the correct headers, many requests will be resolved before even reaching the server!
Some of the above mentioned code smells, even if apparently obvious, are in practice not very simple to spot. For loops and content being generated more than once, it should be quite easy, just look at the number of times a certain function is called and draw your conclusions. Identifying data being processed but not used might be harder: you see the traces, and ideally you should think why you are seeing those calls at all, or why you see them in that particular place. That's why a lot of discipline is required: you keep looking at those reports for so long that you wish you could eliminate (violently) as many calls as possible so you don't have to look at them anymore.

Decouple Services

Do not rely on having all the resources available on the same machine. The more you decouple the various services, the easier it is to scale horizontally. Problem is, how to identify the parts to decouple? Well, first of all think about all the services that can be logically separate from the application itself, like all the data sources, content providers, data stores, but also the data-processing routines that are effectively black boxes. Then you might look at the profiler, and see if there's a resource-intensive routine: can you move it to another machine? Can you maybe add -say- a thin RESTful interface around it? If so, then that service can be moved out of your app, and taken care of separately (e.g. with horizontal replication, if it's a data store, or put on a cluster behind a load balancer if it's a data processor).

Profile Under Load

As a last suggestion, it's a good idea to collect profiler data under load, which is probably more representative of the real usage. To collect a random sample of profiler data, you can run a load testing tool (e.g. apache ab, siege, avalanche) and save a XHProf run every 10000 runs, by modifying the included scripts like this:
/usr/share/php5/utilities/xhprof/header.php
$xhprof_on = false;
if (mt_rand(1, 10000) === 1) {
    $xhprof_on = true;
    if (extension_loaded('xhprof')) {
        include_once '/usr/local/lib/php/xhprof_lib/utils/xhprof_lib.php';
        include_once '/usr/local/lib/php/xhprof_lib/utils/xhprof_runs.php';
        xhprof_enable(XHPROF_FLAGS_CPU + XHPROF_FLAGS_MEMORY);
    }
}
/usr/share/php5/utilities/xhprof/footer.php
if ($xhprof_on && extension_loaded('xhprof')) {
    $profiler_namespace = 'myapp';  // namespace for your application
    $xhprof_data = xhprof_disable();
    $xhprof_runs = new XHProfRuns_Default();
    $run_id = $xhprof_runs->save_run($xhprof_data, $profiler_namespace);
 
    // url to the XHProf UI libraries (change the host name and path)
    $profiler_url = sprintf('http://myhost.com/xhprof/xhprof_html/index.php?run=%s&source=%s', $run_id, $profiler_namespace);
    echo '<a href="'.$profiler_url.'" target="_blank">Profiler output</a>';
}
If your load testing tool can generate reports on CPU and memory usage over time, and collect statistics on what external services are accessed and with what frequency, then by all means observe those graphs, they give a lot of information on the real behaviour of your application and its critical areas. This is a goldmine when it comes to understanding what remains to be optimised. Also make sure the response time remains as flat as possible, without too many spikes or an exponential growth as the load increases: this is a good indicator of stable code and a stable architecture.

Some Parting Thoughts

If you really want to achieve considerable speed gains and scalability improvements, you often have to be ruthless, question everything, ask all the stupid questions, follow the 5 Whys principle and yes, be prepared to annoy everyone else in the team. I think I did that more than once, and I apologise sincerely, but it was in a good cause!

Resources

Some links with more detail about some of the topics mentioned, and some further reading:

http://techportal.ibuildings.com/2009/12/01/profiling-with-xhprof/

Apache Performance Benchmarks

Howto: Performance Benchmarks a Webserver

by nixcraft on June 9, 2006 · 33 comments
You can benchmark Apache, IIS and other web server with apache benchmarking tool called ab. Recently I was asked to performance benchmarks for different web servers.
It is true that benchmarking a web server is not an easy task. From how to benchmark a web server:
First, benchmarking a web server is not an easy thing. To benchmark a web server the time it will take to give a page is not important: you don't care if a user can have his page in 0.1 ms or in 0.05 ms as nobody can have such delays on the Internet.
What is important is the average time it will take when you have a maximum number of users on your site simultaneously. Another important thing is how much more time it will take when there are 2 times more users: a server that take 2 times more for 2 times more users is better than another that take 4 times more for the same amount of users."
Here are few tips to carry out procedure along with an example:

Apache Benchmark Procedures

  • You need to use same hardware configuration and kernel (OS) for all tests
  • You need to use same network configuration. For example, use 100Mbps port for all tests
  • First record server load using top or uptime command
  • Take at least 3-5 readings and use the best result
  • After each test reboot the server and carry out test on next configuration (web server)
  • Again record server load using top or uptime command
  • Carry on test using static html/php files and dynamic pages
  • It also important to carry out test using the Non-KeepAlive and KeepAlive (the Keep-Alive extension to provide long-lived HTTP sessions, which allow multiple requests to be sent over the same TCP connection) features
  • Also don't forget to carry out test using fast-cgi and/or perl tests

Webserver Benchmark Examples:

Let us see how to benchmark a Apache 2.2 and lighttpd 1.4.xx web server.

Static Non-KeepAlive test for Apache web server

i) Note down server load using uptime command
$ uptime
ii) Create a static (small) html page as follows (snkpage.html) (assuming that server IP is 202.54.200.1) in /var/www/html (or use your own webroot):
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>Webserver test</title>
</head>
<body>
This is a webserver test page.
</body>
</html>
 
Login to Linux/bsd desktop computer and type following command:
$ ab -n 1000 -c 5 http://202.54.200.1/snkpage.html
Where,
  • -n 1000: ab will send 1000 number of requests to server 202.54.200.1 in order to perform for the benchmarking session
  • -c 5 : 5 is concurrency number i.e. ab will send 5 number of multiple requests to perform at a time to server 202.54.200.1
For example if you want to send 10 request, type following command:
$ ab -n 10 -c 2 http://www.somewhere.com/
Output:
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.cyberciti.biz (be patient).....done
Server Software:
Server Hostname:        www.somewhere.com
Server Port:            80
Document Path:          /
Document Length:        16289 bytes
Concurrency Level:      1
Time taken for tests:   16.885975 seconds
Complete requests:      10
Failed requests:        0
Write errors:           0
Total transferred:      166570 bytes
HTML transferred:       162890 bytes
Requests per second:    0.59 [#/sec] (mean)
Time per request:       1688.597 [ms] (mean)
Time per request:       1688.597 [ms] (mean, across all concurrent requests)
Transfer rate:          9.59 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      353  375  16.1    386     391
Processing:  1240 1312  52.1   1339    1369
Waiting:      449  472  16.2    476     499
Total:       1593 1687  67.7   1730    1756
Percentage of the requests served within a certain time (ms)
  50%   1730
  66%   1733
  75%   1741
  80%   1753
  90%   1756
  95%   1756
  98%   1756
  99%   1756
 100%   1756 (longest request)
Repeat above command 3-5 times and save the best reading.

Static Non-KeepAlive test for lighttpd web server

First, reboot the server:
# reboot
Stop Apache web server. Now configure lighttpd and copy /var/www/html/snkpage.html to lighttpd webroot and run the command (from other linux/bsd system):
$ ab -n 1000 -c 5 http://202.54.200.1/snkpage.html
c) Plot graph using Spreadsheet or gnuplot.

How do I carry out Web server Static KeepAlive test?

Use -k option that enables the HTTP KeepAlive feature using ab test tool. For example:
$ ab -k -n 1000 -c 5 http://202.54.200.1/snkpage.html
Use the above procedure to create php, fast-cgi and dynmic pages to benchmarking the web server.
Please note that 1000 request is a small number you need to send bigger (i.e. the hits you want to test) requests, for example following command will send 50000 requests :
$ ab -k -n 50000 -c 2 http://202.54.200.1/snkpage.html

How do I save result as a Comma separated value?

Use -e option that allows to write a comma separated value (CSV) file which contains for each percentage (from 1% to 100%) the time (in milliseconds) it took to serve that percentage of the requests:
$ ab -k -n 50000 -c 2 -e apache2r1.cvs http://202.54.200.1/snkpage.html

How do I import result into excel or gnuplot programs so that I can create graphs?

Use above command or -g option as follows:
$ ab -k -n 50000 -c 2 -g apache2r3.txt http://202.54.200.1/snkpage.html
Put following files in your webroot (/var/www/html or /var/www/cgi-bin) directory. Use ab command.

Sample test.php file

#!/usr/bin/perl
$command=`perl -v`;
$title = "Perl Version";
 
print "Content-type: text/html\n\n";
print "<html><head><title>$title</title></head>\n<body>\n\n";
 
print "
<h1>$title</h1>
 
\n";
print $command;
 
print "\n\n</body></html>";
 
Run ab command as follows:
$ ab -n 3000 -c 5 http://202.54.200.1/cgi-bin/test.pl

Sample psql.php (php+mysql) file

<html>
<head><title>Php+MySQL</title></head>
<body>
<?php
   $link = mysql_connect("localhost", "USERNAME", "PASSWORD");
   mysql_select_db("DATABASE");
 
   $query = "SELECT * FROM TABLENAME";
   $result = mysql_query($query);
 
   while ($line = mysql_fetch_array($result))
   {
      foreach ($line as $value)
       {
         print "$value\n";
      }
   }
 
    mysql_close($link);
?>
</body>
</html>
 
Run ab command as follows:
$ ab -n 1000 -c 5 http://202.54.200.1/psql.php

http://www.cyberciti.biz/tips/howto-performance-benchmarks-a-web-server.html

rsync

Task : Copy file from a local computer to a remote server

Copy file from /www/backup.tar.gz to a remote server called openbsd.nixcraft.in
$ rsync -v -e ssh /www/backup.tar.gz jerry@openbsd.nixcraft.in:~Output:
Password:
sent 19099 bytes  received 36 bytes  1093.43 bytes/sec
total size is 19014  speedup is 0.99
Please note that symbol ~ indicate the users home directory (/home/jerry).

Task : Copy file from a remote server to a local computer

Copy file /home/jerry/webroot.txt from a remote server openbsd.nixcraft.in to a local computer /tmp directory:
$ rsync -v -e ssh jerry@openbsd.nixcraft.in:~/webroot.txt /tmp
Password

Task: Synchronize a local directory with a remote directory

$ rsync -r -a -v -e "ssh -l jerry" --delete openbsd.nixcraft.in:/webroot/ /local/webroot

Task: Synchronize a remote directory with a local directory

$ rsync -r -a -v -e "ssh -l jerry" --delete /local/webroot openbsd.nixcraft.in:/webroot

Task: Synchronize a local directory with a remote rsync server

$ rsync -r -a -v --delete rsync://rsync.nixcraft.in/cvs /home/cvs

Task: Mirror a directory between my "old" and "new" web server/ftp

You can mirror a directory between my "old" (my.old.server.com) and "new" web server with the command (assuming that ssh keys are set for password less authentication)
$ rsync -zavrR --delete --links --rsh="ssh -l vivek" my.old.server.com:/home/lighttpd /home/lighttpd


http://www.cyberciti.biz/tips/linux-use-rsync-transfer-mirror-files-directories.html

Monday 20 February 2012

Sql Import

All Loads / Any Source

If you have more than 1GB of data to load, or if your data is coming from somewhere other than a MySQL database, we recommend creating flat files and loading them with mysqlimport. mysqlimport is another command line utility bundled with the MySQL client software whose purpose is to load flat files into MySQL.

We also recommend creating DB Snapshots of the target Amazon RDS DB Instance before and after the data load. Amazon RDS DB Snapshots are complete backups of your DB Instance that can be used to restore your DB Instance to a known state. When you initiate a DB Snapshot, I/O operations to your database instance are temporarily suspended for a few minutes while your database is backed up.


Creating a DB Snapshot immediately before the load allows you restore the database to its state prior to the load, should the need arise. A DB Snapshot taken immediately after the load protects you from having to load the data again in case of a mishap and can also be used to seed new database instances.


be "sales.csv" or "sales.txt", but not "sales_01.csv".  


http://aws.amazon.com/articles/2933

Amazon Relational Database Service

  1. Open a command prompt and enter the following command; make sure to use the endpoint of the DB Instance you created.
    PROMPT> mysql -h mydbinstance.c7hszkfowzmc.us-east-1.rds.amazonaws.com -P 3306 -u mymasteruser -p
    You will see output similar to the following.
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 350
    Server version: 5.1.50-log MySQL Community Server (GPL)
    
    Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
    
    mysql>
     
    http://docs.amazonwebservices.com/AmazonRDS/latest/GettingStartedGuide/ConnectToDBInstance.html 

install php apc on mac

 sudo pear update-channels
 sudo brew install pcre
 sudo pecl install apc

APC is 5 times faster than Memcached

http://stackoverflow.com/questions/1794342/memcache-vs-apc-for-a-single-server-site-data-caching


 
Google say that APC is 5 times faster than Memcached, my experience say that APC is nearly 7-8 times faster than Memcached.. but, memcanched can be accessed by different services (for example, if you run mainly on apache and delegates some traffic, e.g. static contents like images or pure html, to another web-service, like lighttpd), that can be really usefull, if not indispensable.
APC have less feature than memcached and is easly to use and optimize, but this depends on your needs.


If (and only if) you have a web application which has to live on different webservers (loadbalancing), you have to use memcache for distributed caching. If not, just stick to APC and its cache

Sunday 19 February 2012

How to install cron without shell editor

How to install cron without shell editor

Let’s say you don’t have an editor setup to work with crontab on a chroot server and you want to install a cron job.
What you have to do is the following:
$ contab -l
This will print currently installed cron jobs
$ crontab -l > cron.txt
Put contents of the crontab in a file. You can edit this file using and ftp/scp program or by other simple editor available on the server.
When you are ready install the cron…
$ crontab < cron.txt
If no errors are printed, cron job is installed successfully.

http://www.geeksonwork.com/how-to-install-cron-without-shell-editor.html

Cron

     A field may be an asterisk (*), which always stands for ``first-last''.

     Ranges of numbers are allowed.  Ranges are two numbers separated with a hyphen.  The specified range is inclusive.  For example, 8-11 for an ``hours'' entry specifies execution at hours 8, 9, 10 and
     11.

     Lists are allowed.  A list is a set of numbers (or ranges) separated by commas.  Examples: ``1,2,5,9'', ``0-4,8-12''.

     Step values can be used in conjunction with ranges.  Following a range with ``/<number>'' specifies skips of the number's value through the range.  For example, ``0-23/2'' can be used in the hours
     field to specify command execution every other hour (the alternative in the V7 standard is ``0,2,4,6,8,10,12,14,16,18,20,22'').  Steps are also permitted after an asterisk, so if you want to say
     ``every two hours'', just use ``*/2''.

     Names can also be used for the ``month'' and ``day of week'' fields.  Use the first three letters of the particular day or month (case does not matter).  Ranges or lists of names are not allowed.

     The ``sixth'' field (the rest of the line) specifies the command to be run.  The entire command portion of the line, up to a newline or % character, will be executed by /bin/sh or by the shell spec-
     ified in the SHELL variable of the cronfile.  Percent-signs (%) in the command, unless escaped with backslash (\), will be changed into newline characters, and all data after the first % will be
     sent to the command as standard input.  The command can optionally be prefixed by ``@AppleNotOnBattery '' to tell cron not to run the command when functioning on battery power.  For example, the
     ``sixth'' field when using this option would appear something like ``@AppleNotOnBattery /usr/bin/touch /tmp/foo''

     Note: The day of a command's execution can be specified by two fields -- day of month, and day of week.  If both fields are restricted (ie, are not *), the command will be run when either field
     matches the current time.  For example, ``30 4 1,15 * 5'' would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.

     Instead of the first five fields, one of eight special strings may appear:

           string          meaning
           ------          -------
           @reboot         Run once, at startup.
           @yearly         Run once a year, "0 0 1 1 *".
           @annually       (same as @yearly)
           @monthly        Run once a month, "0 0 1 * *".
           @weekly         Run once a week, "0 0 * * 0".
           @daily          Run once a day, "0 0 * * *".
           @midnight       (same as @daily)
           @hourly         Run once an hour, "0 * * * *".

EXAMPLE CRON FILE
     # use /bin/sh to run commands, overriding the default set by cron
     SHELL=/bin/sh
     # mail any output to `paul', no matter whose crontab this is
     MAILTO=paul
     #
     # run five minutes after midnight, every day
     5 0 * * *       $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
     # run at 2:15pm on the first of every month -- output mailed to paul
     15 14 1 * *     $HOME/bin/monthly
     # run at 10 pm on weekdays, annoy Joe
     0 22 * * 1-5    mail -s "It's 10pm" joe%Joe,%%Where are your kids?%
     23 0-23/2 * * * echo "run 23 minutes after midn, 2am, 4am ..., everyday"
     5 4 * * sun     echo "run at 5 after 4 every sunday"

SEE ALSO
     crontab(1), cron(8), launchd.plist(5), launchctl(1), launchd(8)

EXTENSIONS
     When specifying day of week, both day 0 and day 7 will be considered Sunday.  BSD and ATT seem to disagree about this.

     Lists and ranges are allowed to co-exist in the same field.  "1-3,7-9" would be rejected by ATT or BSD cron -- they want to see "1-3" or "7,8,9" ONLY.

     Ranges can include "steps", so "1-9/2" is the same as "1,3,5,7,9".

     Names of months or days of the week can be specified by name.

     Environment variables can be set in the crontab.  In BSD or ATT, the environment handed to child processes is basically the one from /etc/rc.

     Command output is mailed to the crontab owner (BSD cannot do this), can be mailed to a person other than the crontab owner (SysV cannot do this), or the feature can be turned off and no mail will be
     sent at all (SysV cannot do this either).

     All of the `@' commands that can appear in place of the first five fields are extensions.

AUTHORS
     Paul Vixie <paul@vix.com>

BUGS
     If you are in one of the 70-odd countries that observe Daylight Savings Time, jobs scheduled during the rollback or advance will be affected.  In general, it is not a good idea to schedule jobs dur-
     ing this period.

     For US timezones (except parts of IN, AZ, and HI) the time shift occurs at 2AM local time.  For others, the output of the zdump(8) program's verbose (-v) option can be used to determine the moment
     of time shift.