OpDemand. One another cloud managment system.

OpDemand
Of course I never use it, but I like they homepage design. :)

Amazon Web Services Blog: Fast Forward - Provisioned IOPS for EBS Volumes

Amazon Web Services Blog: Fast Forward - Provisioned IOPS for EBS Volumes

--vladget

"IfIsEvil" - Nice article describing why you have not use IF at you nginx configuration

IfIsEvil:
Directive if has problems when used in location context, in some cases it doesn't do what you expect but something completely different instead. In some cases it even segfaults. It's generally a good idea to avoid it if possible...

AWS Service Health Dashboard - Jul 11, 2012

AWS Service Health Dashboard - Jul 11, 2012:
в Багдаде все спокойно

Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more | VentureBeat

Amazon cloud outage takes down Netflix, Instagram, Pinterest, & more | VentureBeat:

'via Blog this'

Getting Good IO from Amazon's EBS

Getting Good IO from Amazon's EBS:

Amazon AWS Elastic load balancer and Apache slowloris

I have just  investigated what Amazon AWS Elastic load balancer helps to protect Apache from "slowloris" DDoS attacks...

How it was tested:
I've use this tool for test.
The first test direct to ec2 instance and the second one to elb with one ec2 instance registered on it...

Finally Amazon implemented my idea: using Spot Instances for Auto Scaling!

"This release of Amazon Elastic Compute Cloud (Amazon EC2) introduces several new Spot Instance features that make it easier to get started and handle the potential of interruption. The new features include integration with Auto Scaling and AWS CloudFormation. ..."

Release Notes: http://aws.amazon.com/releasenotes/0814290369059292?ref_=pe_12300_24358190
Docs: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/spot-launching-autoscaling.html

[SOLVED] native fileinfo (version 1.0.2-dev) fails at php 5.3.X

Zend exception:

finfo_open() [function.finfo-open]: Warning: description `8-bit ISDN mu-law compressed (CCITT G.721 ADPCM voice data enco' truncated
/mnt/www/zend/Zend/Validate/File/MimeType.php on line 199
#0 [internal function]: exceptionErrorHandler(8, 'finfo_open() [<...', '/mnt/www/zend/Z...', 199, Array)

I thought it was Zend bug, but bug it native fileinfo extension...

test php script:

php
$finfo = finfo_open(FILEINFO_MIME, "/usr/share/file/magic"); // return mime type ala mimetype extension

if (!$finfo) {
    echo "Opening fileinfo database failed";
    exit();
}
?>

SOLUTION:
Just install fileinfo from pecl:

# cd /tmp# wget http://pecl.php.net/get/Fileinfo-1.0.4.tgz# tar -xzvf Fileinfo-1.0.4.tgz# phpize && ./configure && make# sudo make install 

-- vladget

WTF!? Amazon EC2 is DOWN RIGHT NOW! WTF??!

Automatic incremental MySQL backup with xtrabackup.

#!/bin/bash

now=`date +%Y-%m-%d_%H-%M-%S`
backup_directory="/media/data/backup"
current_backup=$backup_directory/$now"_full"

logfile="/var/log/db-backup/"$now"_full"

mysql_user="root"
mysql_pass=""


echo "========================================="
echo "        MySQL Data Backup (Full)"
echo "========================================="

#perform full backup
innobackupex --ibbackup=xtrabackup_55 --parallel=4 --user=$mysql_user --password=$mysql_pass --no-timestamp $current_backup 2> $logfile

# see if the backup completed sucsessfully
if ! tail $logfile -n 1 | grep completed.OK
then
        rm -R $current_backup
        echo "\nERROR: backup failed! (log: "$logfile")"
  exit 1
fi

echo $current_backup > $backup_directory/last_backup

echo "Backup Completed OK!"

MySQL: increasing query cache hit rate.

Part 1: Expecting SQL queries.

The most common mistakes:

1. Using different cases

Incoming queries are compared to those in the query cache before parsing, so the following two queries are regarded as different by the query cache:
    SELECT * FROM tbl_name
    Select * from tbl_name

Advice: use database abstraction layer or stick to single coding style.

2. Call to date/time functions at database side

A query cannot be cached if it contains any of the functions shown in the following list:

CONVERT_TZ() 

CURDATE() 

CURRENT_DATE() 

CURRENT_TIME() 

CURRENT_TIMESTAMP() 

CURTIME() 

NOW() 

SYSDATE() 

UNIX_TIMESTAMP() with no parameters 

Advice(UPDATED): Move date/time function to your application and cut off the seconds or/and  minutes  or/and hour part.

(If your database server and your application on the same, or both configured to same timezone, and using ntp servers).

3. Call to RAND(), FOUND_ROWS() functions at database side

Same for previos case: Queries which contain these function cannot be cached.

Advice: You could found "RAND()" analog at any programming language. Use SQL_CALC_FOUND_ROWS() function instead FOUND_ROWS()

4. Merging huge request with tables with high changing rate

If a table changes, all cached queries that use the table become invalid and are removed from the cache. This includes queries that use MERGE tables that map to the changed table. A table can be changed by many types of statements, such as INSERT, UPDATE, DELETE, TRUNCATE TABLE, ALTER TABLE, DROP TABLE, or DROP DATABASE.

Advice: Spit it to 2+ requests. 

Two queries(the first one which contains "huge part", but cached, and the second one for table with high change rate) will work faster then one merged.

For more information: http://dev.mysql.com/doc/refman/5.0/en/query-cache-operation.html

-- Vladimir Getmanshchuk aka vladget

spotscale diagram

src: http://mycld.googlecode.com/files/SpotScale.svg

Elastician: Comprehensive List of AWS Endpoints

endpoints cheat-sheet

Elastician: Comprehensive List of AWS Endpoints:

'via Blog this'

Bugtraq: AdaCore Security Advisory SA-2012-L119-003 Hash collisions in AWS

Bugtraq: AdaCore Security Advisory SA-2012-L119-003 Hash collisions in AWS:

'via Blog this'

--vladget

mycld: spotscale - autoscaling solution for Amazon EC2 based on Spot Instances.


I've commited new debuged version today.
Please review: http://code.google.com/p/mycld/source/browse/

My Clouding exp.: mycld: spotscale - autoscaling solution for Amazon...

My Clouding exp.: mycld: spotscale - autoscaling solution for Amazon...: mycld: spotscale - autoscaling solution for Amazon EC2 based on Spot Instances. I've commited new debuged version today. Please review:...

New version of mycld released!


New features:
- OOP
- Logger (see the log dir: tail -f /root/mycld/logs/*.log)
- implemented: all adjustment types, works with more then one elb, something else, i dont remember all :)
- fixed: a lot of bugs

How to disable IPv6 at FreeBSD 9.x


add to /etc/rc.conf:

# IPv6 Networking
ipv6_network_interfaces="none"          # Default is auto
ipv6_activate_all_interfaces="NO"       # this is the default
ip6addrctl_enable="NO"                  # New way to disable IPv6 support
ip6addrctl_policy="ipv4_prefer"         # Use IPv4 instead of IPv6

more info here

-- vladget (Vladimir Getmanshchuk)

The Paranoiac's feeds:

The first one and the bigest one.
Feed List:
Security Mailing List Archive

Addition feeds:
National Vulnerability Database (NIST USA)
US-CERT Current Activity
Amazon Linux AMI Security Bulletins
FreeBSD ports collection

   

NoSQL from AWS called "Dynamo". LOL.



Amazon DynamoDB - Internet-Scale Data Storage the NoSQL Way
Posted: 18 Jan 2012 09:32 AM PST
We want to make it very easy for you to be able to store any amount of semistructured data and to be able to read, write, and modify it quickly, efficiently, and with predictable performance. We don't want you to have to worry about servers, disks, replication, failover, monitoring, software installation, configuration, or updating, hardware upgrades, network bandwidth, free space, sharding, rearchitecting, or a host of other things that will jump up and bite you at the worst possible time.

We want you to think big, to dream big dreams, and to envision (and then build) data-intensive applications that can scale from zero users up to tens or hundreds of millions of users before you know it. We want you to succeed, and we don't want your database to get in the way. Focus on your app and on building a user base, and leave the driving to us.

Sound good?


from aws blog

NEW VERSION 0.0.6 RELEASED!
POST UPDATED!

AWS auto-scaling for EC2 spot-instances!
How to implement.

mycld project at code.google.com

Requirements:

You must to have setup AWS auto-scaling or you have to setup it using this guide.
(btw. You need AWS Autoscaling CLI tools for setup your autoscaling.)

Make sure what your autoscalig configuration works correct on on-demand ec2 instances.
So your autoscaling configuration must have:
- lauch-config at least with options: --image-id value --instance-type value --group value[,value...] --key value, other options are optinal.
[--block-device-mapping "key1=value1,key2=value2..." ]  - still did not implemented!

- auto-scaling-group at least with options: --availability-zones value[,value...] --launch-configuration value --max-size value --min-size value --load-balancers value[, value]], other options are optinal.
[--desired-capacity value] [--grace-period value] [--health-check-type value][--placement-group value] [--vpc-zone-identifier value] -still did not implemented

- policies mandatory options: --auto-scaling-group --adjustment= --type --cooldown

- alarms: all except one: value of option: --dimension, can't be autoscaling metric like 
--dimensions "AutoScalingGroupName=MyAutoScalingGroup", because cloudwatch has no data for current scaling process.
I recommend You use loadbalancer metric, or use my common design for autoscaling: one on-demand instance at master role + scaleout  with spot-instance in slave role.



Also you need *nix like OS (FreeBSD, Linux, Mac OS X, etc) with:

 - PHP 5.3.x +
 - MySQL 5.x
 - AWSSDKforPHP (and all its dependents)
 - PEAR PHP Log

Installation:

1. Install dependencies

1.1 Make sure what PEAR has been installed and path included in include_path directive at your php.ini.
To adding directive to php.ini (where /usr/share/pear - path to you pear):

    # echo "include_path=\".:/usr/share/pear\"" >> /etc/php.ini

1.2 Discover pear channel and install AWSSDKforPHP(aka cloudfusion):

    # pear channel-discover pear.amazonwebservices.com
    # pear install aws/sdk

1.3 Config SDK:

    # cp -Rp /usr/share/pear/AWSSDKforPHP/config-sample.inc.php ~/.aws/sdk/config.inc.php
    # vim ~/.aws/sdk/config.inc.php (add your AWS security credentials to config)
    # chmod 600 ~/.aws/sdk/config.inc.php

1.4. Install Log package from PEAR:

    # pear install Log

 

2. Install mycld

2.1 Get the mycld from code.google.com

    # svn checkout http://mycld.googlecode.com/svn/trunk ~/mycld

2.2 Create MySQL database

   # mysql -u root -p < ~/mycld/db.sql

2.3 Config mycld for accessing your MySQL

    # cd ~/mycld/conf/
    # cp -Rp  db.ini.sample  db.ini
    # vim db.ini (change your mysql user and/or password)

Running:

1. Suspend you AWS autoscaling activities first:

    # as-suspend-processes MyAutoScalingGroup

2. Add cron job:

    # echo "*/5 * * * * root /root/mycld/mycld.cli.runner.php \

-s spotscale -a check -d >> /root/mycld/logs/spotscale-error.log \

2>&1" >> /etc/crontab 

Debugging:



1. Check what you are running mycld.cli.runner.php with "-d"'.

    # grep "mycld" /etc/crontab | grep "-d"

2. Watch logs:
 

    # tail -f ~/mycld/logs/*.log

How to implement AWS auto-scaling for spot-instances

I've described it at mycld blog:
http://mycld.blogspot.com/2012/01/aws-auto-scaling-for-ec2-spot-instances.html

-- vladget

Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data

http://alestic.com/2010/01/ec2-instance-locking

Searching through the AWS SDK for PHP documentation
http://blog.ryanparman.com/2011/02/27/searching-through-the-aws-sdk-for-php-documentation/

Really usefull for AWS developming for PHP!