Sunday 28 February 2016

Apache Thrift ™

Ruby Tutorial


Introduction

All Apache Thrift tutorials require that you have:
  1. Built and installed the Apache Thrift Compiler and Libraries, see Building from source for more details.
  2. Generated the tutorial.thrift and shared.thrift files as discussed here.
    thrift -r --gen rb tutorial.thrift
    
  3. Followed all prerequisites listed below.

Prerequisites

Client

$:.push('gen-rb')
$:.unshift '../../lib/rb/lib'

require 'thrift'

require 'calculator'

begin
  port = ARGV[0] || 9090

  transport = Thrift::BufferedTransport.new(Thrift::Socket.new('localhost', port))
  protocol = Thrift::BinaryProtocol.new(transport)
  client = Calculator::Client.new(protocol)

  transport.open()

  client.ping()
  print "ping()\n"

  sum = client.add(1,1)
  print "1+1=", sum, "\n"

  sum = client.add(1,4)
  print "1+4=", sum, "\n"

  work = Work.new()

  work.op = Operation::SUBTRACT
  work.num1 = 15
  work.num2 = 10
  diff = client.calculate(1, work)
  print "15-10=", diff, "\n"

  log = client.getStruct(1)
  print "Log: ", log.value, "\n"

  begin
    work.op = Operation::DIVIDE
    work.num1 = 1
    work.num2 = 0
    quot = client.calculate(1, work)
    puts "Whoa, we can divide by 0 now?"
  rescue InvalidOperation => io
    print "InvalidOperation: ", io.why, "\n"
  end

  client.zip()
  print "zip\n"

  transport.close()

rescue Thrift::Exception => tx
  print 'Thrift::Exception: ', tx.message, "\n"
end

Server

$:.push('gen-rb')
$:.unshift '../../lib/rb/lib'

require 'thrift'

require 'calculator'
require 'shared_types'

class CalculatorHandler
  def initialize()
    @log = {}
  end

  def ping()
    puts "ping()"
  end

  def add(n1, n2)
    print "add(", n1, ",", n2, ")\n"
    return n1 + n2
  end

  def calculate(logid, work)
    print "calculate(", logid, ", {", work.op, ",", work.num1, ",", work.num2,"})\n"
    if work.op == Operation::ADD
      val = work.num1 + work.num2
    elsif work.op == Operation::SUBTRACT
      val = work.num1 - work.num2
    elsif work.op == Operation::MULTIPLY
      val = work.num1 * work.num2
    elsif work.op == Operation::DIVIDE
      if work.num2 == 0
        x = InvalidOperation.new()
        x.whatOp = work.op
        x.why = "Cannot divide by 0"
        raise x
      end
      val = work.num1 / work.num2
    else
      x = InvalidOperation.new()
      x.whatOp = work.op
      x.why = "Invalid operation"
      raise x
    end

    entry = SharedStruct.new()
    entry.key = logid
    entry.value = "#{val}"
    @log[logid] = entry

    return val

  end

  def getStruct(key)
    print "getStruct(", key, ")\n"
    return @log[key]
  end

  def zip()
    print "zip\n"
  end

end

handler = CalculatorHandler.new()
processor = Calculator::Processor.new(handler)
transport = Thrift::ServerSocket.new(9090)
transportFactory = Thrift::BufferedTransportFactory.new()
server = Thrift::SimpleServer.new(processor, transport, transportFactory)

puts "Starting the server..."
server.serve()
puts "done."

Thursday 18 February 2016

Redis Cheat Sheet

source: http://lzone.de/cheat-sheet/Redis


Redis Cheat Sheet

When you encounter a Redis instance and you quickly want to learn about the setup you just need a few simple commands to peak into the setup. Of course it doesn't hurt to look at the official full command documentation, but below is a listing just for sysadmins.

Accessing Redis

CLI

First thing to know is that you can use "telnet" (usually on default port 6397)
telnet localhost 6397
or the Redis CLI client
redis-cli
to connect to Redis. The advantage of redis-cli is that you have a help interface and command line history.

CLI Queries

Here is a short list of some basic data extraction commands:
TypeSyntax and Explanation
TracingWatch current live commands. Use this with care on production. Cancel with Ctrl-C.
monitor
Slow Queries
slowlog get 25  # print top 25 slow queries
slowlog len  
slowlog reset
Search Keys
keys pattern  # Find key matching exactly
keys pattern*  # Find keys matching in back
keys *pattern*  # Find keys matching somewhere
keys pattern*  # Find keys matching in front
On production servers use "KEYS" with care as it causes a full scan of all keys!
Generic
del <key>
dump <key>  # Serialize key
exists <key>
expire <key> <seconds>
Scalars
get <key> 
set <key> <value>
setnx <key> <value> # Set key value only if key does not exist
Batch commands:
mget <key> <key> ...
mset <key> <value> <key> <value> ...
Counter commands:
incr <key>
decr <key>
Lists
lrange <key> <start> <stop>
lrange mylist 0 -1  # Get all of a list
lindex mylist 5   # Get by index
llen mylist   # Get length

lpush mylist "value"
lpush mylist 5   
rpush mylist "value"

lpushx mylist 6   # Only push in mylist exists
rpushx mylist 0 

lpop mylist
rpop mylist

lrem mylist 1 "value"  # Remove 'value' count times
lset mylist 2 6   # mylist[2] = 6
ltrim <key> <start> <stop>
Hashes
hexists myhash field1  # Check if hash key exists

hget myhash field1
hdel myhash field2
hset myhash field1 "value"
hsetnx myhash field1 "value"

hgetall myhash
hkeys myhash
hlen myhash
Batch commands:
hmget <key> <key> ...
hmset <key> <value> <key> <value> ...
Counter commands
hincrby myhash field1 1
hincrby myhash field1 5
hincrby myhash field1 -1

hincrbrfloat myhash field2 1.123445 
SetsFIXME
Sorted SetsFIXME

CLI Scripting

For scripting just pass commands to "redis-cli". For example:
$ redis-cli INFO | grep connected
connected_clients:2
connected_slaves:0
$

Server Statistics

The statistics command is "INFO" and will give you an output as following.
$ redis-cli INFO
redis_version:2.2.12
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:8353
uptime_in_seconds:2592232
uptime_in_days:30
lru_clock:809325
used_cpu_sys:199.20
used_cpu_user:309.26
used_cpu_sys_children:12.04
used_cpu_user_children:1.47
connected_clients:2   # <---- connection count
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:6596112
used_memory_human:6.29M   # <---- memory usage
used_memory_rss:17571840
mem_fragmentation_ratio:2.66
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1371241671
bgrewriteaof_in_progress:0
total_connections_received:118
total_commands_processed:1091
expired_keys:441
evicted_keys:0
keyspace_hits:6
keyspace_misses:1070
hash_max_zipmap_entries:512
hash_max_zipmap_value:64
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:master    # <---- master/slave in replication setup
db0:keys=91,expires=88

Changing Runtime Configuration

The command
CONFIG GET *
gives you a list of all active configuration variables you can change. The output might look like this:
redis 127.0.0.1:6379> CONFIG GET *
 1) "dir"
 2) "/var/lib/redis"
 3) "dbfilename"
 4) "dump.rdb"
 5) "requirepass"
 6) (nil)
 7) "masterauth"
 8) (nil)
 9) "maxmemory"
10) "0"
11) "maxmemory-policy"
12) "volatile-lru"
13) "maxmemory-samples"
14) "3"
15) "timeout"
16) "300"
17) "appendonly"
18) "no"
19) "no-appendfsync-on-rewrite"
20) "no"
21) "appendfsync"
22) "everysec"    # <---- how often fsync() is called
23) "save"
24) "900 1 300 10 60 10000"  # <---- how often Redis dumps in background
25) "slave-serve-stale-data"
26) "yes"
27) "hash-max-zipmap-entries"
28) "512"
29) "hash-max-zipmap-value"
30) "64"
31) "list-max-ziplist-entries"
32) "512"
33) "list-max-ziplist-value"
34) "64"
35) "set-max-intset-entries"
36) "512"
37) "slowlog-log-slower-than"
38) "10000"
39) "slowlog-max-len"
40) "64"
Note that keys and values are alternating and you can change each key by issuing a "CONFIG SET" command like:
CONFIG SET timeout 900
Such a change will be effective instantly. When changing values consider also updating the redis configuration file.

Databases

Multiple Databases

Redis has a concept of separated namespaces called "databases". You can select the database number you want to use with "SELECT". By default the database with index 0 is used. So issuing
redis 127.0.0.1:6379> SELECT 1
OK
redis 127.0.0.1:6379[1]>
switches to the second database. Note how the prompt changed and now has a "[1]" to indicate the database selection. To find out how many databases there are you might want to run redis-cli from the shell:
$ redis-cli INFO | grep ^db
db0:keys=91,expires=88
db1:keys=1,expires=0

Dropping Databases

To drop the currently selected database run
FLUSHDB
to drop all databases at once run
FLUSHALL

Replication

Checking for Replication

To see if the instance is a replication slave or master issue
redis 127.0.0.1:6379> INFO
[...]
role:master
and watch for the "role" line which shows either "master" or "slave". Starting with version 2.8 the "INFO" command also gives you per slave replication status looking like this
slave0:ip=127.0.0.1,port=6380,state=online,offset=281,lag=0

Setting up Replication

If you quickly need to set up replication just issue
SLAVEOF <IP> <port>
on a machine that you want to become slave of the given IP. It will immediately get values from the master. Note that this instance will still be writable. If you want it to be read-only change the redis config file (only available in most recent version, e.g. not on Debian). To revert the slave setting run
SLAVEOF NO ONE


Performance Testing

Benchmark

Install the Redis tools and run the provided benchmarking tool
redis-benchmark -h <host> [-p <port>]
If you are migrating from/to memcached protocol check out how to run the same benchmark for any key value store with memcached protocol.

Debugging Latency

First measure system latency on your Redis server with
redis-cli --intrinsic-latency 100
and then sample from your Redis clients with
redis-cli --latency -h <host> -p <port>
If you have problems with high latency check if transparent huge pages are disabled. Disable it with
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Dump Database Backup

As Redis allows RDB database dumps in background, you can issue a dump at any time. Just run:
BGSAVE
When running this command Redis will fork and the new process will dump into the "dbfilename" configured in the Redis configuration without the original process being blocked. Of course the fork itself might cause an interruption. Use "LASTSAVE" to check when the dump file was last updated. For a simple backup solution just backup the dump file. If you need a synchronous save run "SAVE" instead of "BGSAVE".

Listing Connections

Starting with version 2.4 you can list connections with
CLIENT LIST
and you can terminate connections with
CLIENT KILL <IP>:<port>

Monitoring Traffic

The propably most useful command compared to memcached where you need to trace network traffic is the "MONITOR" command which will dump incoming commands in real time.
redis 127.0.0.1:6379> MONITOR
OK
1371241093.375324 "monitor"
1371241109.735725 "keys" "*"
1371241152.344504 "set" "testkey" "1"
1371241165.169184 "get" "testkey"
additionally use "SLOWLOG" to track the slowest queries in an interval. For example
SLOWLOG RESET
# wait for some time
SLOWLOG GET 25
and get the 25 slowest command during this time.

Friday 12 February 2016

make sense?

Victor Nava You know a few months ago, I was working on a program to solve permutations and combinations.

This is a very interesting problem because you can think of many things in terms of combinations or sequences. Put this right thing in the right place and it makes something. Somehow, Music is a sequence of notes, books are sequences of letters, a painting is a sequence of material.

This somehow got me thinking about infinity. What is infinity, is the universe infinite? is anything infinite? is infinity possible?

Then I stopped thinking about that.

A few months later I was working on different program that generated drawings. A sequence of pixels on the screen.

Than I got thinking about combinations and infinity again. You know when you are not working you have time to think about silly things.

I asked my self, how is it possible that a program can make a drawing? well that's easy, it is possible because I am telling the program what to do. Cool. Then I thought what if the program could make drawings with random parameters until it makes one that I like? umm that sounds possible.

Than I though, what don't I just make a really dumb program that generates ALL of the possible drawings I can fit on my screen. Shit that would take a long time, but just for fun, how many drawings can in theory be drawn on my computer screen?

My initial answer was: a infinite number.

That can't be right, if the number of pixels on my screen is finite and the number of colors that each pixel can be in at any given time is finite, how is it possible to draw an infinite number of pictures?

Let's see, my screen has a resolution of 1200 by 900 pixels. That means that there are in total about 1 Million (1200 * 900) pixels. Each pixel can have lit in 16777216 different colors.

So how many drawing can be drawn on my screen?

The answer is simple and it comes from combinations.

An image is nothing more that a sequence of numbers, that represent pixels.
Each pixel can be of a particular color. That's it get the right pixel lit up with the right color and you get an picture.

To calculate the number of the possible pictures that can be shown on a screen you take the number of pixels and the number of colors that each pixel can be in and apply the mathematical formula to find the number of combinations of two sets.

The result is a very very very big number. However it is a finite number, which means you can only put so many different pictures on a particular screen.

Hold on, there are so many things I can draw on my screen.

Realising this made me a bit sad.

I started thinking about creativity and imagination and soon realised that this terms as romantic as they sound, obey the same rules. There are only so many things we can think of. Because the number of cells in our brains is finite and thoughts come from our brain, from the combination of signals from neurons firing messages to each other.

So in theory, you could simulate every single cell in a brain, and run a program that calculates every possible combination of connections, and messages. The same as with a computer screen.

This sound impossible, and it is at the moment.

To generate all of the possible pictures on my screen, it would take my computer an enormous amount time. But if you give it enough time it will eventually get there. Now, this is using current technology and dumb algorithms.

The other interesting question is, how many pictures that make sense can be shown on my screen?

Again the number is finite. Because this is a subset of all the possible pictures. So in theory it should be easier to find the answer, right?

No. Doing this is actually really hard. Because to do this you need to tell the computer what a picture that makes sense looks like. And we don't know how to describe that in words. We can tell what makes sense but we can't figure out the formula.

However, you can teach a computer how to find things that make sense by training it. Pretty much the same way you teach humans, by example. You show it a bunch pictures that make sense and a bunch that don't and let it know which one is wish. If you give it enough examples it can "learn" to recognise the pattern. This is already possible, the problem is how to find the right data.

But in theory you can take a computer programm and feed every single web page on the internet, every sms, facebook and whatsapp message, plus all the books ever written. And calculate the right combination of letters to form words, sentences and stories that make sense. Within the number of stories that make sense there will be many that will be superior to any other ever written.

Pause here for a moment. This programs are not taught how to write, they are taught how to recognise patterns that humans like and emulate them. To the computers this patterns are just numbers. Not pixels, or letters just numbers.

There are a finite number of letter in the alphabet and a finite number of letters you can put in a 400 pages of paper.

A book is a combination of sentences.
A sentence is a combination of words.
A word is a combination of letters.
All finite. And predictable.

Where the ideas to combine letters in an way that makes sense comes from is a mystery.

But when you can see all possibilities you don't need to imagine. You just need to select.

From all of the possible books that can be written lie the works of Dostoevsky. Computers will eventually generate all of them, plus the ones he didn't have time to write.

They will be able to imitate the style of writers. And combine styles from different people to form new styles. And we won't be able to tell the difference. Or maybe we will, we'll say things like: "this book is too good, it must have been generated by some stupid program".

to be continued...





Me:
Where the ideas to combine letters in an way that makes sense comes from is a mystery.

sense: A sense is a physiological capacity of organisms that provides data for perception. The senses and their operation, classification, and theory are overlapping topics studied by a variety of fields, most notably neuroscience, cognitive psychology (or cognitive science), and philosophy of perception

So: If we can map out what make sense to us and we can formalize this somehow so the computer can understand they we can solve this mystery

But how we can do this?

Sense: to me its just a sequence of the mind electricity transitions that its happening in our mind.
But what formula we can use here to save it when "something makes sense" to us so the computers can understand?

Another thought:
Is this the right approache? Or do we have to save to computer what are the facts about the elements around us and what is this that we want so a situation wil make sense. What i mean is:

Is it the "make sense" a process based on existing defined truths and the results that we want?

results = :foo

defined_truths = [4,2,3,5]

is the process make sense

defined_truths.include? results

the actually "include?" method when its happening do we do the same with our minds?

So: is it the "make sense" a sequence of examples that they can work? and it can aply to all the stuff in life?

results = i wanna fly
defined_truths = [gravity, weight]

the make sense

If i wanna fly the gravity should be less that x
Can i do this? yes ok makes sense

So for me its an array of examples that can be true to nature laws and the laws we define smile emoticon

and again it comes something else:

Who can define laws and why and how? Based again in a array of valid and accepted examples?

Big story and very interesting grin emoticon




i was walking now and i was thinking that we have to DEFINE everything to the computer so its common sense to them also.  What it is each Object and what aretheir attributes so we can compare select etc




Tuesday 9 February 2016

Stripe::CountrySpec

My contribution to stripe-gem





Now you can use



https://github.com/stripe/stripe-ruby/commit/f8532d225e20cb6ded2bd9a672a6d8a0479b80ba