Admin Linux ... MySQL DBA

MySQL : Altering Huge Tables

2010-12-07T02:31:00.000-08:00

You have a huge mysql table - maybe 100 GB. And you need to run alter on it - to either add an index, drop an index, add a column or drop a column. If you run the simple mysql "alter table" command, you will end up spending ages to bring the table back into production.

Here is a simple hack to get the thing done. The benefit of the hack is that the alter runs quite fast. But since this is a hack, you will need to take care of the backups - in case anything goes wrong. I have seen this hack work effectively with both MyISAM and InnoDB tables.

Here I have created a simple table to show this hack process. You can assume that this table has billions of rows and is more than 100GB in size.

CREATE TABLE `testhack` (
`id` int(11) NOT NULL DEFAULT '0',
`unq` varchar(100) DEFAULT NULL,
`keyword` varchar(250) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unq` (`unq`)
) ENGINE=MyISAM

I need to drop the unique key. So, i have create a new table 'testhack_new' with the following schema

CREATE TABLE `testhack_new` (
`id` int(11) NOT NULL DEFAULT '0',
`unq` varchar(100) DEFAULT NULL,
`keyword` varchar(250) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM

Flush both tables with read lock

mysql> Flush tables with read lock;

Open another terminal. And go to the mysql/data/ directory. Do the following:

mysql/data/test $ mv testhack.frm testhack_old.frm; mv testhack_new.frm testhack.frm; mv testhack_old.frm testhack_new.frm; mv testhack.MYI testhack_old.MYI; mv testhack_new.MYI testhack.MYI; mv testhack_old.MYI testhack_new.MYI;

So, what is happening here is that the index, table definitions are being switched. After this process, the table definition of testhack will not contain the unique key. Now unlock the tables in the main window. And run repair tables to remove any issues.

mysql> unlock tables;
mysql> repair tables testhack;

+---------------+--------+----------+-------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+---------------+--------+----------+-------------------------------------------+
| test.testhack | repair | warning | Number of rows changed from 0 to 20000000 |
| test.testhack | repair | status | OK |
+---------------+--------+----------+-------------------------------------------+

The repair table rebuilds the indexes. It is faster since it skips the use of key_cache for rebuilding the index (which is used in a normal alter table scenario).

Source :
http://www.mysqlperformanceblog.com/2007/10/29/hacking-to-make-alter-table-online-for-certain-changes/

mysql-packages

2010-09-13T02:13:00.000-07:00

The RPM packages usages and details:

Ø MySQL-server-VERSION.glibc23.i386.rpm - The MySQL server. You need this unless you only want to connect to a MySQL server running on another machine.

Ø MySQL-client-VERSION.glibc23.i386.rpm - The standard MySQL client programs. You probably always want to install this package.

Ø MySQL-devel-VERSION.glibc23.i386.rpm - The libraries and include files that are needed if you want to compile other MySQL clients, such as the Perl modules.

Ø MySQL-debuginfo-VERSION.glibc23.i386.rpm - This package contains debugging information. debuginfo RPMs are never needed to use MySQL software; this is true both for the server and for client programs. However, they contain additional information that might be needed by a debugger to analyze a crash.

Ø MySQL-shared-VERSION.glibc23.i386.rpm - This package contains the shared libraries (libmysqlclient.so*) that certain languages and applications need to dynamically load and use MySQL. It contains single-threaded and thread-safe libraries. If you install this package, do not install the MySQL-shared-compat package.

Ø MySQL-shared-compat-VERSION.glibc23.i386.rpm - This package includes the shared libraries for MySQL 3.23, 4.0, and so on, up to the current release. It contains single-threaded and thread-safe libraries. Install this package instead of MySQL-shared if you have applications installed that are dynamically linked against older versions of MySQL but you want to upgrade to the current version without breaking the library dependencies.

Ø MySQL-shared-compat-advanced-gpl-VERSION.glibc23.i386.rpm, MySQL-shared-compat-advanced-VERSION.glibc23.i386.rpm - These are like the MySQL-shared-compat package, but are for the “MySQL Enterprise Server – Advanced Edition” products. Install these packages rather than the normal MySQL-shared-compat package if you want to included shared client libraries for older MySQL versions.

Ø MySQL-embedded-VERSION.glibc23.i386.rpm - The embedded MySQL server library.
Ø MySQL-ndb-management-VERSION.glibc23.i386.rpm, MySQL-ndb-storage-VERSION.glibc23.i386.rpm, MySQL-ndb-tools-VERSION.glibc23.i386.rpm, MySQL-ndb-extra-VERSION.glibc23.i386.rpm - Packages that contain additional files for MySQL Cluster installations.
o Note : The MySQL-ndb-tools RPM requires a working installation of perl. Prior to MySQL 5.1.18, the DBI and HTML::Template packages were also required. See Section 2.15, “Perl Installation Notes”, and Section 17.4.21, “ndb_size.pl — NDBCLUSTER Size Requirement Estimator”, for more information.

Ø MySQL-test-VERSION.glibc23.i386.rpm - This package includes the MySQL test suite.
Ø MySQL-VERSION.src.rpm - This contains the source code for all of the previous packages. It can also be used to rebuild the RPMs on other architectures (for example, Alpha or SPARC).

Default Installation Directory :

Directory -> Contents of Directory
/usr/bin -> Client programs and scripts
/usr/sbin -> The mysqld server
/var/lib/mysql -> Log files, databases
/usr/share/info -> Manual in Info format
/usr/share/man -> Unix manual pages
/usr/include/mysql -> Include (header) files
/usr/lib/mysql -> Libraries
/usr/share/mysql -> Miscellaneous support files, including error messages, character set files, sample configuration files, SQL for database installation
/usr/share/sql-bench -> Benchmarks

Security Issues with MySQL ROOT Access

2010-06-27T05:48:00.000-07:00

Security Issues with MySQL ROOT Access

MySQL offers simple but very effective security mechanisms. Unfortunately, the default installation of MySQL, and in particular the empty root password and the potential vulnerability to buffer overflow attacks, makes the database an easy target for attacks.
In order to achieve the highest possible level of security, the installation and configuration of MySQL should be performed in accordance with the following security requirements:

* MySQL processes must run under a unique UID/GID that is not used by any other system process.
* Only local access to MySQL need to be allowed.(some exceptions for jobs/backups)
* MySQL root's account must be protected by a complex/hard to guess password.
* The administrator's account (root) need to be renamed.
* Anonymous access to the database (by using the nobody account) must be disabled.

MySQL Security risks can be categorized into the following.

* Filesystem security risks. MySQL Installation (basedir) and database information (datadir) and other log (querylog/slowlog) files that contain information about queries that clients execute. These files/directories need to be protected so that other users who have login accounts on the server host cannot access them directly.
* Network security risks. The MySQL server provides access to databases by allowing clients to connect and make requests. Information about client accounts is stored in the mysql database. Each account should be set up with privileges that provide access only to the data the accounts needs to see or modify.

MySQL root account has full privileges to perform any database operation, so it's important to assign the account a password that is not easily guessed. Note that usernames and passwords for MySQL accounts are unrelated to those for system login accounts. OS login and MySQL login both should not be the same.

Restrict anonymous remote access

Grant access to specific users from specific hosts only. Do not grant access from all hosts.
Do not grant the PROCESS or SUPER privilege to non-administrative users. The output of mysqladmin processlist and SHOW PROCESSLIST shows the statements currently being executed, so any user who is allowed to see the server process list might be able to see statements issued by other users such as UPDATE user SET password=PASSWORD(pwd).

mysqld reserves an extra connection for users who have the SUPER privilege, so that a MySQL root user can log in and check server activity even if all normal connections are in use. This is very useful when MySQL reaches the max_connections threshold. This cannot be beneficial; if users root (which ever has SUPER privilege) is used for all client connections.
Do not grant the FILE privilege to non-administrative users. Any user that has this privilege can write a file anywhere in the file system with the privileges of the mysqld daemon. To make this a bit safer, files generated with SELECT ... INTO OUTFILE do not overwrite existing files and are writable by everyone.
The FILE privilege may also be used to read any file that is accessible to the Unix user that the server runs as. With this privilege, user can read any file into a database table. This could be abused, for example, by using LOAD DATA to load /etc/passwd into a table, which then can be displayed with SELECT.

Improve local security

Use different socket file for both client and server connections. The following parameter should be changed in the [client] section of /etc/my.cnf:
[client] socket = /tmp/mysql.sock

Change admin password

One of the most important steps in securing MySQL is changing the database administrator's password, which is empty by default. In order to change the administrator's password, follow the steps:
mysql -u root mysql> SET PASSWORD FOR root@localhost=PASSWORD('new_password');
It is good practice not to change passwords from the command line,(instead change at database level). This is especially important when other users working on the server. In that case the password could be easily revealed, e.g. by using the "ps aux" command or reviewing history files, when improper access rights are set to them.

Change admin name

It is also recommended to change the default name of administrator's account (root), to a different, harder to guess one. Such a change will make it difficult to perform brute-force and dictionary attacks on the administrator's password.
mysql> update user set user="mysqluser" where user="root"; mysql> flush privileges;

Remove history

We should also remove the content of the MySQL history file (~/.mysql_history, ~/.history, ~/.bash_history,~/.mysql_history) in which all executed SQL commands are being stored (especially passwords, which are stored as plain text).
User Access Privileges
We can create accounts for specific databases which will be used by specific applications. These accounts should have access rights only to the databases which are used by the specific applications. In particular, they should not have any access rights to the mysql database, nor any system or administrative privileges (FILE, GRANT, ALTER, SHOW DATABASE, RELOAD, SHUTDOWN, PROCESS, SUPER etc.). Application users should not granted all privileges to database with Grant option from any host.

mysql> select user,host,password from user; GRANT USAGE ON *.* TO 'user1'@'%' IDENTIFIED BY PASSWORD 'xxxxxx' | GRANT ALL PRIVILEGES ON `user1`.* TO 'glist'@'%' WITH GRANT OPTION

MySQL Development History

2010-06-25T00:05:00.000-07:00

MySQL Development History

- MySQL was first released internally on 23 May 1995
- Windows version was released on January 8, 1998 for Windows 95 and NT
- Version 3.23: beta from June 2000, production release January 2001
- Version 4.0: beta from August 2002, production release March 2003
- Version 4.1: beta from June 2004, production release October 2004
- Version 5.0: beta from March 2005, production release October 2005
- Version 5.1: currently pre-production (since November 2005)
- Sun Microsystems acquires MySQL AB on 26 February 2008

MySQL Features History

- Version 3.23.23 Full-Text Search
- Version 4.0 Full-Text Search (IN BOOLEAN MODE), UNIONS
- Version 4.1 R-Tree and B-Tree, Sub-Queries, Prepared Statements
- Version 5.0 Cursors, Stored Procedures, Triggers, Views, XA Transactions
- Version 5.1 Event Scheduler, Partitioning, Plugin API, Row-Based Replication, Server Log Tables

Display Information About File - Stat

2010-06-13T11:51:00.000-07:00

In Unix/Linux world everything is treated as files. whether it is a devices, directories and sockets — all of these are files.

Stat command displays file or file system status.

[Prabhat@Server1 Archive]$ stat 1_16470_587807474.arc
File: `1_16470_587807474.arc'
Size: 208514560 Blocks: 407664 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 17006596 Links: 1
Access: (0640/-rw-r-----) Uid: ( 500/ oracle) Gid: ( 500/ dba)
Access: 2010-06-12 23:28:58.000000000 -0700
Modify: 2010-06-12 23:31:22.000000000 -0700
Change: 2010-06-12 23:31:22.000000000 -0700

Details of Linux Stat Command Output

* File: `1_16470_587807474.arc’ – Name of the file.
* Size: 208514560 – File size in bytes.
* Blocks: 407664 – Total number of blocks used by this file.
* IO Block: 4096 – IO block size for this file.
* regular file – Indicates the file type. This indicates that this is a regular file. Following are available file types.

o regular file. ( ex: all normal files ).
o directory. ( ex: directories ).
o socket. ( ex: sockets ).
o symbolic link. ( ex: symbolic links. )
o block special file ( ex: hard disk ).
o character special file. ( ex: terminal device file ).

* Device: fd02h/64770d – Device number in hex and device number in decimal
* Inode: 17006596 – Inode number is a unique number for each file which is used for the internal maintenance by the file system.
* Links: 1 – Number of links to the file
* Access: (0640/-rw-r-----): Access specifier displayed in both octal and character format. Let us see explanation about both the format.
* Uid: ( 500/oracle) – File owner’s user id and user name are displayed.
* Gid: ( 500/dba) – File owner’s group id and group name are displayed.
* Access: 2010-06-12 23:28:58.000000000 -0700 – Last access time of the file.
* Modify: 2010-06-12 23:31:22.000000000 -0700 – Last modification time of the file.
* Change: 2010-06-12 23:31:22.000000000 -0700 – Last change time of the inode data of that file.

Stat – Display Information About Directory

You can use the same command to display the information about a directory as shown below.

[Prabhat@Server1 oradata]$ stat Archive
File: `Archive'
Size: 12288 Blocks: 24 IO Block: 4096 directory
Device: fd02h/64770d Inode: 16990209 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 500/ oracle) Gid: ( 500/ dba)
Access: 2009-11-02 23:43:22.000000000 -0800
Modify: 2010-06-13 05:26:54.000000000 -0700
Change: 2010-06-13 05:26:54.000000000 -0700

MySQL : About LIMIT 0 , 30

2010-06-03T00:28:00.000-07:00

Limit is used to limit your MySQL query results to those that fall within a specified range. You can use it to show the first X number of results, or to show a range from X - Y results. It is phrased as Limit X, Y and included at the end of your query.
X is the starting point (remember the first record is 0) and Y is the duration (how many records to display).

Table 'example' have following data.
SELECT * FROM `example` ORDER BY `example`.`age` ASC LIMIT 0 , 30

id name age
3 Amit 26
5 pani 27
2 Sanjay 28
4 Lucky 29
6 atul 30
1 Kumar 31

SELECT * FROM `example` ORDER BY `example`.`age` ASC LIMIT 3 , 1

id name age
4 Lucky 29

This will show records 3rd only (remember the first record is 0)

MySQL : Allocating memory for caches

2010-05-27T22:21:00.001-07:00

Allocating memory for caches in MySQL

If the server is dedicated to MySQL, any memory you don’t reserve for the operating system or for query processing is available for caches.
MySQL needs more memory for caches than anything else. It uses caches to avoid disk access, which is orders of magnitude slower than accessing data in memory. The operating system may cache some data on MySQL’s behalf (especially for MyISAM),but MySQL needs lots of memory for itself too.

The following are the most important caches to consider for the majority of installations:
• The operating system caches for MyISAM data
• MyISAM key caches
• The InnoDB buffer pool
• The query cache
There are other caches, but they generally don’t use much memory.

It is much easier to tune a server if you’re using only one storage engine. If you’re using only MyISAM tables, you can disable InnoDB completely, and if you’re using only InnoDB, you need to allocate only minimal resources for MyISAM (MySQL uses MyISAM tables internally for some operations). But if you’re using a mixture of storage engines, it can be very hard to figure out the right balance between them.

The MyISAM Key Cache

The MyISAM key caches are also referred to as key buffers; there is one by default, but you can create more. Unlike InnoDB and some other storage engines, MyISAM itself caches only indexes, not data (it lets the operating system cache the data). If you use mostly MyISAM, you should allocate a lot of memory to the key caches.

The most important option is the key_buffer_size, which you should try setting to between 25% and 50% of the amount of memory you reserved for caches. The remainder will be available for the operating system caches, which the operating system will usually fill with data from MyISAM’s .MYD files.

MySQL 5.0 has a hard upper limit of 4 GB for this variable, no matter what architecture you’re running. (MySQL 5.1 allows larger sizes. Check the current documentation for your version of the server.)

By default MyISAM caches all indexes in the default key buffer, but you can create multiple named key buffers. This lets you keep more than 4 GB of indexes in memory at once. To create key buffers named key_buffer_1 and key_buffer_2, each sized at 1 GB, place the following in the configuration file:
key_buffer_1.key_buffer_size = 1G
key_buffer_2.key_buffer_size = 1G
Now there are three key buffers: the two explicitly created by those lines and the default buffer. You can use the CACHE INDEX command to map tables to caches. You can also tell MySQL to use key_buffer_1 for the indexes from tables t1 and t2 with the following SQL statement:
mysql> CACHE INDEX t1, t2 IN key_buffer_1;
Now when MySQL reads blocks from the indexes on these tables, it will cache the blocks in the specified buffer. You can also preload the tables’ indexes into the cache with the LOAD INDEX command:
mysql> LOAD INDEX INTO CACHE t1, t2;

You can place this SQL into a file that’s executed when MySQL starts up. The filename must be specified in the init_file option, and the file can include multiple SQL commands, each on a single line (no comments are allowed). Any indexes you don’t explicitly map to a key buffer will be assigned to the default buffer the first time MySQL needs to access the .MYI file.

You can monitor the performance and usage of the key buffers with information from SHOW STATUS and SHOW VARIABLES. You can calculate the hit ratio and the percentage of the buffer in use with these equations:
Cache hit ratio
100 - ( (Key_reads * 100) / Key_read_requests )
Percentage of buffer in use
100 - ( (Key_blocks_unused * key_cache_block_size) * 100 / key_buffer_size )

It’s good to know the cache hit rate, but this number can be misleading. For example,the difference between 99% and 99.9% looks small, but it really represents a tenfold increase. The cache hit rate is also application-dependent: some applications might work fine at 95%, whereas others might be I/O-bound at 99.9%. You might even be able to get a 99.99% hit rate with properly sized caches.

The number of cache misses per second is generally much more empirically useful. Suppose you have a single hard drive that can do 100 random reads per second. Five misses per second will not cause your workload to be I/O-bound, but 80 per second
will likely cause problems.
You can use the following equation to calculate this value cache misses:
Key_reads / Uptime
Calculate the number of misses incrementally over intervals of 10 to 100 seconds, so you can get an idea of the current performance. The following command will show the incremental values every 10 seconds:
$ mysqladmin extended-status -r -i 10 | grep Key_reads
When you’re deciding how much memory to allocate to the key caches, it might help to know how much space your MyISAM indexes are actually using on disk. You don’t need to make the key buffers larger than the data they will cache.
Space your MyISAM indexes are actually using on disk
$ du -sch `find /path/to/mysql/data/directory/ -name "*.MYI"`

Remember that MyISAM uses the operating system cache for the data files, which are often larger than the indexes. Therefore, it often makes sense to leave more memory for the operating system cache than for the key caches. Finally, even if you don’t have any MyISAM tables, bear in mind that you still need to set key_buffer_size to a small amount of memory, such as 32M. The MySQL server sometimes uses MyISAM tables for internal purposes, such as temporary tables for GROUP BY queries.

The MyISAM key block size The key block size is important (especially for write-intensive workloads) because of the way it causes MyISAM, the operating system cache, and the filesystem to interact. If the key block size is too small, you may encounter read-around writes, which
are writes that the operating system cannot perform without first reading some data from the disk. Here’s how a read-around write happens, assuming the operating system’s page size is 4 KB (typically true on the x86 architecture) and the key block size is 1 KB:
1. MyISAM requests a 1 KB key block from disk.
2. The operating system reads 4 KB of data from the disk and caches it, then passes the desired 1 KB of data to MyISAM.
3. The operating system discards the cached data in favor of some other data.
4. MyISAM modifies the 1 KB key block and asks the operating system to write it back to disk.
5. The operating system reads the same 4 KB of data from the disk into the operating system cache, modifies the 1 KB that MyISAM changed, and writes the entire
4 KB back to disk.

The read-around write happened in step 5, when MyISAM asked the operating system to write only part of a 4 KB page. If MyISAM’s block size had matched the operating system’s, the disk read in step 5 could have been avoided.*Unfortunately, in MySQL 5.0 and earlier, there’s no way to configure the key block size. However, in MySQL 5.1 and later, you can avoid read-around writes by making MyISAM’s key block size the same as the operating system’s. The myisam_block_size variable controls the key block size. You can also specify the size for each key with the KEY_BLOCK_SIZE option in a CREATE TABLE or CREATE INDEX statement, but because all keys are stored in the same file, you really need all of them to have blocks as large as or larger than the operating system’s to avoid alignment issues that could still cause read-around writes. (For example, if one key has 1 KB blocks and another has 4 KB blocks, the 4 KB block boundaries might not match the operating system’s page boundaries.)

The InnoDB Buffer Pool

If you use mostly InnoDB tables, the InnoDB buffer pool probably needs more memory than anything else. Unlike the MyISAM key cache, the InnoDB buffer pool doesn’t just cache indexes: it also holds row data, the adaptive hash index, the insert buffer, locks, and other internal structures. InnoDB also uses the buffer pool to help it delay writes, so it can merge many writes together and perform them sequentially. In short, InnoDB relies heavily on the buffer pool, and you should be sure to allocate enough memory to it.

The MySQL manual suggests using up to 80% of the machine’s physical memory for the buffer pool on a dedicated server; in reality, you can use more than that if the machine has a lot of memory.

As with the MyISAM key buffers, you can use variables from SHOW commands or tools such as innotop to monitor your InnoDB buffer pool’s memory usage and performance. There’s no equivalent of LOAD INDEX INTO CACHE for InnoDB tables. However, if you’re trying to warm up a server and get it ready to handle a heavy load, you can issue queries that perform full table scans or full index scans.

In most cases, you should make the InnoDB buffer pool as large as your available memory allows. However, in rare circumstances, very large buffer pools (say, 50 GB) can cause long stalls. For example, a large buffer pool may become slow during checkpoints or insert buffer merge operations, and concurrency can drop as a result of locking. If you experience these problems, you may have to reduce the buffer pool size.

You can change the innodb_max_dirty_pages_pct variable to instruct InnoDB to keep more or fewer dirty (modified) pages in the buffer pool. If you allow a lot of dirty pages, InnoDB can take a long time to shut down, because it writes the dirty pages to the data files upon shutdown.
You can monitor the number of dirty pages by watching the Innodb_buffer_pool_pages_dirty server status variable or using innotop to monitor SHOW INNODB STATUS.
Lowering the value of the innodb_max_dirty_pages_pct variable doesn’t actually guarantee that InnoDB will keep fewer dirty pages in the buffer pool. Instead, it controls the threshold at which InnoDB stops being “lazy.” InnoDB’s default behavior is to flush dirty pages with a background thread, merging writes together and performing them sequentially for efficiency. This behavior is called “lazy” because it lets InnoDB delay flushing dirty pages in the buffer pool, unless it needs to use the space for some other data. When the percentage of dirty pages exceeds the threshold, InnoDB will flush pages as quickly as it can to try to keep the dirty page count lower. The variable’s default value is 90, so by default InnoDB will flush lazily until the buffer pool is 90% full of dirty pages. You can tweak the threshold for your workload if you wish to spread out the writes a bit more. For example, lowering it to 50 will generally cause InnoDB to do more write operations, because it will flush pages sooner and therefore be unable to batch the writes as well. However, if your workload has a lot of write spikes, using a lower value may help InnoDB absorb the spikes better: it will have more “spare” memory to hold dirty pages, so it won’t have to wait for other dirty pages to be flushed to disk.

The Thread Cache

The thread cache holds threads that aren’t currently associated with a connection but are ready to serve new connections. When there’s a thread in the cache and a new connection is created, MySQL removes the thread from the cache and gives it to the new connection. When the connection is closed, MySQL places the thread back into the cache, if there’s room. If isn’t room, MySQL destroys the thread. As long as MySQL has a free thread in the cache, it can respond very rapidly to connect requests, because it doesn’t have to create a new thread for each connection.
The thread_cache_size variable specifies the number of threads MySQL can keep in the cache. You probably won’t need to tune this value, unless your server gets many connection requests. To check whether the thread cache is large enough, watch the Threads_created status variable. We generally try to keep the thread cache large enough that we see fewer than 10 new threads created each second, but it’s often pretty easy to get this number lower than 1 per second.

A good approach is to watch the Threads_connected variable and try to set thread_cache_size large enough to handle the typical fluctuation in your workload. For example, if Threads_connected usually stays between 100 and 200, you can set the cache size to 100. If it stays between 500 and 700, a thread cache of 200 should be large enough. Think of it this way: at 700 connections, there are probably no threads in the cache; at 500 connections, there are 200 cached threads ready to be used if the load increases to 700 again. Making the thread cache very large is probably not necessary for most uses, but keeping it small doesn’t save much memory, so there’s little benefit in doing so. Each thread that’s in the thread cache or sleeping typically uses around 256 KB of memory. This is very little compared to the amount of memory a thread can use when a connection is actively processing a query. In general, you should keep your thread cache large enough that Threads_created doesn’t increase very often. If this is a very large number, however (e.g., many thousand threads), you might want to set it lower because some operating systems don’t handle very large numbers of threads well, even when most of them are sleeping.

The Table Cache

The table cache is similar in concept to the thread cache, but it stores objects that represent tables. Each object in the cache contains the associated table’s parsed .frm file, plus other data. Exactly what else is in the object depends on the table’s storage engine. For example, for MyISAM, it holds the table data and/or index file descriptors. For merge tables it may hold many file descriptors, because merge tables can have many underlying tables. The table cache can help you reuse resources. For instance, when a query requests access to a MyISAM table, MySQL might be able to give it a file descriptor from the cached object instead of opening the file. The table cache can also help avoid some of the I/O required for marking a MyISAM table as “in use” in the index headers.*The table cache’s design is a little MyISAM-centric—this is one of the areas where the separation between the server and the storage engines is not completely clean, for historical reasons. The table cache is a little less important for InnoDB, because InnoDB doesn’t rely on it for as many purposes (such as holding file descriptors; it has its own version of a table cache for this purpose). However, even InnoDB benefits from caching the parsed .frm files.

In MySQL 5.1, the table cache is separated into two parts: a cache of open tables and a table definition cache (configured via the table_open_cache and table_definition_cache variables). Thus, the table definitions (the parsed .frm files) are separated from
the other resources, such as file descriptors. Opened tables are still per-thread, pertable-used, but the table definitions are global and can be shared among all connections efficiently. You can generally set table_definition_cache high enough to cache all your table definitions. Unless you have tens of thousands of tables, this is likely to be the easiest approach. If the Opened_tables status variable is large or increasing, the table cache isn’t large enough, and you should increase the table_cache system variable (or table_open_cache, in MySQL 5.1). The only real downside to making the table cache very large is that it might cause longer shutdown times when your server has a lot of MyISAM tables, because the key blocks have to be flushed and the tables have to be marked as no longer open. It can also make FLUSH TABLES WITH READ LOCK take a long time complete, for the same reason.
If you get errors indicating that MySQL can’t open any more files (use the perror utility to check what the error number means), you might also need to increase the number of files MySQL is allowed to keep open. You can do this with the open_files_limit server variable in your my.cnf file.
The thread and table caches don’t really use much memory, and they are beneficial
because they conserve resources. Although creating a new thread and opening a new
file aren’t really expensive compared to other things MySQL might do, the overhead
can add up quickly under a high-concurrency workload. Caching threads and tables
can improve efficiency.

MYSQL : Checking and Reparing tables

2010-05-25T09:05:00.001-07:00

If your database is MySQL, though, there's still hope. Over the course of this article, I will show you how to use MySQL's built-in crash recovery tools to check your database, and hopefully recover all the data you just lost.

Built-in tools
When it comes to repairing and checking tables, MySQL offers two options:

myisamchk designed specifically to check and repair MyISAM tables (the default table type used by MySQL). This tool can scan your databases, identify problems, optimize tables for faster lookup, and optionally repair corrupted tables. The myisamchk tool is invoked from the command line.
MySQL also allows you check and repair tables using SQL commands. The CHECK TABLE, REPAIR TABLE, and OPTIMIZE TABLE commands can be used on a running MySQL server and need to be entered through a MySQL client.

Most of the time, it's preferable to use myisamchk because it is significantly faster than using SQL commands.

Checking tables
If you're having trouble accessing a table, first try checking it for errors. To check a table, shut down the server and type myisamchk at the command prompt, followed by the table's file name, as shown below:
$ myisamchk /usr/local/mysql/data/phpbb/user.myi
Checking MyISAM file: /usr/local/mysql/data//user.MYI
Data records: 1153 Deleted blocks: 0
- check file-size
- check key delete-chain
- check record delete-chain
- check index reference
- check data record references index: 1
- check record links

Use the complete path to the table file instead of just the table name. Remember to include the file extension as well.

myisamchk options :
myisamchk -- fast Perform a quick check, only verifying if the table handler closed successfully
myisamchk -- medium-check Perform a faster check, by calculating a checksum for the indexes in each record and verifying that checksum against the index tree
myisamchk -- extend-check Perform a thorough check of the table, verifying the data in each record
Command line arguments for myisamchk

Repairing tables

Next, shut down the MySQL server. This is because myisamchk makes changes to the table file itself, so you don't want other users accessing or otherwise manipulating the file during the repair operation. Also advisable table physical backup of database.

Once the table file is backed up and the server down, you can run myisamchk with the --recover option, as shown below:
$ myisamchk --recover /usr/local/mysql/data/phpbb/user.myi
- recovering (with sort) MyISAM-table '/usr/local/mysql/data/phpbb/user.MYI'Data records: 1153
- Fixing index 1

The --recover option reconstructs the MySQL table index file after weeding the corrupted table of invalid or previously deleted records that could be causing a problem.

In the unlikely event that the --recover option fails, revert to the original table file and try the --safe-recover option. This is slower, because MySQL needs to scan through the records in the data file one by one and them restore the index, but it can sometimes work better than a regular repair operation.
$ myisamchk --safe-recover /usr/local/mysql/data/phpbb/user.myi
- recovering (with keycache) MyISAM-table '/usr/local/mysql/data/phpbb/user.MYI'
Data records: 4

Using CHECK TABLE and REPAIR TABLE

check and repair operations are run from a MySQL prompt, via the CHECK TABLE and REPAIR TABLE commands.

Consider the following example of running a CHECK TABLE command:
mysql> CHECK TABLE user;
+--------------+-------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+--------------+-------+----------+----------+
| phpbb.user | check | status | OK |
+--------------+-------+----------+----------+
1 row in set (0.02 sec)

You can add the FAST, MEDIUM, and EXTENDED keywords to the command to obtain behavior similar to that available in myisamchk. For example:
mysql> CHECK TABLE user EXTENDED;
+--------------+-------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+--------------+-------+----------+----------+
| phpbb.user| check | status | OK |
+--------------+-------+----------+----------+
1 row in set (0.02 sec)

When it comes to repairing tables, too, MySQL offers the REPAIR TABLE option, as below:
mysql> REPAIR TABLE user;
+--------------+--------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+--------------+--------+----------+----------+
| phpbb.user | repair | status | OK |
+--------------+--------+----------+----------+
1 row in set (0.01 sec)

As with the myisamchk tool, the REPAIR TABLE command can take one of two additional options: QUICK, which tries a quick repair, and EXTENDED, which rebuilds the index after reading each record in the table.

An option here is to use the mysqlcheck utility, which provides a command line front end to the CHECK TABLE and REPAIR TABLE commands, allowing you to check and repair tables without first taking the server down. Everything that you can do with CHECK TABLE and REPAIR TABLE can be done with mysqlcheck, by passing appropriate command line options to the program.

MySQL : Calculate Query Executed per second

2010-05-08T07:39:00.000-07:00

Here is a way to calculate Query Executed per second :

time=10
begin=`mysql -e "show status" | awk '{if ($1 == "Questions") print $2}'`
sleep $time
end=`mysql -e "show status" | awk '{if ($1 == "Questions") print $2}'`
diff=`expr $end - $begin`
avg=`expr $diff / $time`
echo "$avg"

Problem : CTRL+S in Putty

2010-04-07T00:30:00.000-07:00

Over the years, we all have habit of using CTRL+S every few minutes during working on a document, because we all had too much work lost from stupid errors; In the Windows world, CTRL+S is used as the Save you work.
But, this habit will be a problem on working in the Linux world.

By accident while inside a terminal window (in PUTTY) we press CTRL+S, this accidental keystroke meant we had reconnect to my Linux server, kill whatever program we were running, and then start it again.

But here is solution :

CTRL+S actually does XOFF, which means the terminal will accept key strokes but won’t show the output of anything. It will appear as if your terminal is dead when it’s really just waiting to be turned back on. The fix? Simply press CTRL+Q to turn flow-control on (XON). If you pressed a whole bunch of keys before pressing CTRL+Q, you’ll see the output from those keystrokes.

memory-usage-free

2010-03-21T22:21:00.000-07:00

There is simple command to check for the overall usage of memory including physical memory and swap memory.

What you is just type free and it will shows you the information of the memory usage. for example:

total used free shared buffers cached
Mem: 16442272 16355460 86812 0 129600 10120444
-/+ buffers/cache: 6105416 10336856
Swap: 10223608 263608 9960000

The value shows at the first line of total column, is the total physical memory I have in kilobytes. I have 16GB ram, the results (16442272kb) shown above is approximate in kb. So the system have already used almost all the physically memory, which just left about 84M as free memory.

Out of 15.6G from 5.8G that stored in the system, is actually used. And it still left 9.8G in cache. I have total 10G of swap , and it uses 9.4GM and remain 9.4G unused.

The memory usage seems high but still consider healthy. I have loaded a lots of applications and uses some minor disc space for swap, but everything still running smooth without any lags.

Free have few options, if you feel that memory result shows in kb is not detail enough, you can have more accurate measurement in bytes. You can passes -b for bytes, or less details -m for Megabytes, -g for gigabytes.

free -b

You can ask free to display results in every 5 seconds, in order to track the increases/decreases on memory usage.

free -s 5

Kontrollbase-MySQL monitoring tool

2010-02-19T06:27:00.000-08:00

Kontrollbase - MySQL monitoring tool.
Past few days I am looking for a centralized MySQL database monitoring tool, Which will give complete details of MySQL database.
I have looked various tools but, Kontrollbase is given me the complete solution.

Kontrollbase is a monitoring, analytics, reporting, and historical analysis webapp for MySQL database administrators and advanced users of MySQL databases.

Important links:
software-kontrollbase
screenshots
demo Username: demo , Password: password
note: do not use IE, use Firefox or other browser.
downloads
Documentation

System details of Windows

2010-02-15T00:34:00.000-08:00

Various time we required system (on Windows 2000, XP, 2003/8, Vista and Windows 7) details like hardware (motherboard, CPU , RAM , etc), installed software operating system’s security hotfixes and lots and lots of more.

One such solution is LookInMyPC – a free program that automatically generates a detailed audit report of the hardware and software on your computer. It records essential information such as, operating system and processor details, the amount of RAM installed, drive specifications, and much, much, more.

sample report

MySQL Tips : Verify database Objects

2009-12-31T16:31:00.000-08:00

Verify database Objects:

1. Verify table.
a. SHOW TABLE STATUS LIKE '%user%'\G
b. SHOW TABLE STATUS from forum LIKE '%user%' \G

2. It will show you all tables have MyISAM engine.
a. SELECT TABLE_SCHEMA, table_name, table_type, engine From information_schema.tables where engine='MyISAM';

3. To know rows per table, you can use a query like this:
a. SELECT table_name, table_rows FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'forum';

4. Verify you procedure / function.
a. SHOW procedure/FUNCTION STATUS LIKE ´hello´G
b. show create procedure/function hello\G

5. Routines
a. select routine_name from information_schema.routines; // list of all the routines in the system
b. select routine_name, routine_schema, routine_type from information_schema.routines; // This lists all of the routines in the system. With additional Information like the database the routines belongs too and also distinct between the different routines.
c. select routine_name, routine_schema, routine_type from information_schema.routines where routine_schema = database() // This time we can see just the routines for the currently selected database.

6. Triggers
a. select TRIGGER_SCHEMA,TRIGGER_NAME from INFORMATION_SCHEMA.TRIGGERS;
b. select TRIGGER_NAME from INFORMATION_SCHEMA.TRIGGERS where TRIGGER_SCHEMA = database();

MySQL Tips : Calculate database and table size

2009-12-28T16:35:00.000-08:00

Few MySQL Tips:

a. calculate databases size :
-> It will show database size

SELECT s.schema_name, CONCAT(IFNULL(ROUND(SUM(t.data_length)/1024/1024,2),0.00),"Mb") as Data_size, CONCAT(IFNULL(ROUND(SUM(t.index_length)/1024/1024,2),0.00),"Mb") as Index_size,COUNT(table_name) total_tables FROM INFORMATION_SCHEMA.SCHEMATA s LEFT JOIN INFORMATION_SCHEMA.TABLES t ON s.schema_name = t.table_schema WHERE s.schema_name not in("mysql","information_schema","test") GROUP BY s.schema_name order by Data_size DESC;

-> It will show database size along with table size.

SELECT s.schema_name,t.table_name, CONCAT(IFNULL(ROUND(SUM(t.data_length)/1024/1024,2),0.00),"Mb") data_size,CONCAT(IFNULL(ROUND(SUM(t.index_length)/1024/1024,2),0.00),"Mb") index_size, t.ENGINE ENGINE, t.table_rows TABLE_ROWS,t.row_format TABLE_ROW_FORMAT,date(t.update_time) FROM INFORMATION_SCHEMA.SCHEMATA s LEFT JOIN INFORMATION_SCHEMA.TABLES t ON s.schema_name = t.table_schema WHERE s.schema_name not in ("mysql","information_schema") GROUP BY s.schema_name,t.table_name,TABLE_ROW_FORMAT,ENGINE ORDER BY TABLE_ROWS DESC,data_size DESC,index_size DESC;

-> It will show table size of engine 'MEMORY'.

SELECT concat(table_schema,'.',table_name) as Database_Tablename, table_rows as Rows, concat(round(data_length/(1024*1024),2),'M') DATA, concat(round(index_length/(1024*1024),2),'M') idx, concat(round((data_length+index_length)/(1024*1024),2),'M') total_size, round(index_length/data_length,2) idxfrac FROM information_schema.TABLES where ENGINE='MEMORY';

-> It will show 5 top tables using space.

SELECT concat(table_schema,'.',table_name) as Database_Tablename,
table_rows as Rows, concat(round(data_length/(1024*1024),2),'M') DATA,
concat(round(index_length/(1024*1024),2),'M') idx, concat(round((data_length+index_length)/(1024*1024),2),'M') total_size,
round(index_length/data_length,2) idxfrac FROM information_schema.TABLES where ORDER BY data_length+index_length DESC limit 5;

Temporarily stop/start a process in linux

2009-12-23T02:05:00.000-08:00

Some time we have requirement that, particular job should stop for certain period of time and start again.

Most of us familiar with KILL command, but here is another feature of KILL command , which saves your life :-

#kill -STOP 10067 (where 10067 is process id)

#kill -CONT 10067 (where 10067 is process id)

have a fun :)

Extract single db/table from dump file

2009-11-04T00:25:00.000-08:00

using sed to extract single database from the dumpfile:
=========================================
sed -n '/^-- Current Database: `dbname`/,/^-- Current Database: `/p' dumpfile > dbname.sql 2>error

eg:
sed -n '/^-- Current Database: `blogs`/,/^-- Current Database: `/p' dump.sql > blogs.sql 2>error

we can also use "awk" for the same.

extract single table from the dumpfile:-

nohup : commands keep executing even you exit from a shell prompt

2009-10-27T16:42:00.000-07:00

nohup command if added in front of any command will continue running the command or process even if you shut down your terminal or close your session to machine

nohup command-name &

Where,

* command-name : is name of shell script or command name. You can pass argument to command or a shell script.
* & : nohup does not automatically put the command it runs in the background; you must do that explicitly, by ending the command line with an & symbol.

examples:

# nohup mysql -q -uUSER1 -pPASS1 < dump.sql > dump.log 2> error.log &

source:
http://www.idevelopment.info/data/Unix/General_UNIX/GENERAL_RunningUNIXCommandsImmunetoHangups_nohup.shtml

http://www.cyberciti.biz/tips/nohup-execute-commands-after-you-exit-from-a-shell-prompt.html

Reset your MySQL Table Auto Increment Value

2009-10-07T20:33:00.000-07:00

Recently,I created tables using auto increment number as the Primary Key. As name say, the number will increase every time I insert new record into the table. At some point I wanted to delete the whole dummy data I inserted.

ALTER TABLE tablename AUTO_INCREMENT = value;

eg.
If I had a items table and I created a few new items and then deleted them, to set the auto increment value back to ‘100′ i would simply:

ALTER TABLE items AUTO_INCREMENT = 100;

MySQL log file rotation

2009-09-13T02:27:00.000-07:00

Rotating MySQL Log Files on Linux.
A. Check logrotate must on daily cron
#less /etc/cron.daily/logrotate

B. Check on logrotate.d there must be mysql
#less /etc/logrotate.d/mysql
Its default Content is:

# by setting the variable "err-log"
# in the [safe_mysqld] section as follows:
#
# [safe_mysqld]
# err-log=/var/lib/mysql/mysqld.log
#
# If the root user has a password you have to create a
# /root/.my.cnf configuration file with the following
# content:
#
# [mysqladmin]
# password =
# user= root
#
# where "" is the password.
#
# ATTENTION: This /root/.my.cnf should be readable ONLY
# for root !

/var/lib/mysql/mysqld.log {
# create 600 mysql mysql
notifempty
daily
rotate 3
missingok
compress
postrotate
# just if mysqld is really running
if test -x /usr/bin/mysqladmin && \
/usr/bin/mysqladmin ping &>/dev/null
then
/usr/bin/mysqladmin flush-logs
fi
endscript
}

Read uses of variable
My file :

# This logname can be set in /etc/my.cnf
# by setting the variable "err-log"
# in the [safe_mysqld] section as follows:
#
# [safe_mysqld]
# err-log=/var/lib/mysql/mysqld.log
#
# If the root user has a password you have to create a
# /root/.my.cnf configuration file with the following
# content:
#
# [mysqladmin]
# password =
# user= root
#
# where "" is the password.
#
# ATTENTION: This /root/.my.cnf should be readable ONLY
# for root !

# - I put everything in one block and added sharedscripts, so that mysql gets
# flush-logs'd only once.
# Else the binary logs would automatically increase by n times every day.
# - The error log is obsolete, messages go to syslog now.
/var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log {
daily
rotate 7
missingok
create 640 mysql adm
compress
sharedscripts
postrotate
test -x /usr/bin/mysqladmin || exit 0

# If this fails, check debian.conf!
export HOME=/etc/mysql/my.cnf
MYADMIN="/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf"
if [ -z "`$MYADMIN ping 2>/dev/null`" ]; then
# Really no mysqld or rather a missing debian-sys-maint user?
# If this occurs and is not a error please report a bug.
if ps cax | grep -q mysqld; then
exit 1
fi
else
$MYADMIN flush-logs
fi
endscript

C. Create a file /root/.my.cnf.
#vi /root/.my.cnf (# ATTENTION: This /root/.my.cnf should be readable ONLY)

Its contain ;

[mysqladmin]
password = xxxx
user= root

Run manually ;

#logrotate -f /path/to/some/logrotate.conf
if you want to force a rotatation which is uncalled for in the config files, you need to give it a "-f":

MySQL: Analyze slow query log using mysqldumpslow

2009-09-09T06:56:00.000-07:00

First enable slow query logging, then generate a slow query and finally look at the slow query log.

mysqldumpslow
This program parses and summarizes a 'slow query log'.
-v verbose
-d debug
-s=WORD
what to sort by (t, at, l, al, r, ar etc)
-r reverse the sort order (largest last instead of first)
-t=NUMBER
just show the top n queries
-a don't abstract all numbers to N and strings to 'S'
-n=NUMBER
abstract numbers with at least n digits within names
-g=WORD
grep: only consider stmts that include this string
-h=WORD
hostname of db server for *-slow.log filename (can be wildcard)
-i=WORD
name of server instance (if using mysql.server startup script)
-l don't subtract lock time from total time
eg.

1. mysqldumpslow -t 10 /var/lib/mysql/mysql-slow.log.1 > /tmp/top_ten_slow_query.txt
It will put top ten slow query in file /tmp/top_ten_slow_query.txt
2. mysqldumpslow -a -s c -t 10 /var/lib/mysql/mysql-slow.log.1 > /tmp/top_ten_repeat_slow_query.txt
It will put top ten repeat slow query in file top_ten_repeat_slow_query.txt

AWK: Pattern Matching and Processing

2009-07-30T12:55:00.000-07:00

awk 'pattern {action}' filename

Reads one line at a time from file, checks for pattern match, performs action if pattern matched pattern.

NR is a special awk variable meaning the line number of the current record

can use a line number, to select a specific line, by comparing it to NR (for example: NR == 2)
can specify a range of line numbers (for example: NR == 2, NR == 4)

can specify a regular expression, to select all lines that match

$n are special awk variables, meaning the value of the nth field (field delimiter is space or tab)

$0 is the entire record
can use field values, by comparing to $n (for example: $3 == 65)

every line is selected if no pattern is specified

Instructions

print - print line(s) that match the pattern, or print fields within matching lines
print is default if no action is specified
there are many, many instruction, including just about all C statements with similar syntax
other instructions will be covered in future courses

examples, using the file testfile.

awk 'NR == 2, NR == 4' testfile - print the 2nd through 4th lines (default action is to print entire line)

awk '/chevy/' testfile - print only lines matching regular expression, same as grep 'chevy' testfile
awk '{print $3, $1}' testfile - print third and first field of all lines (default pattern matches all lines)

awk '/chevy/ {print $3, $1}' testfile - print third and first fiield of lines matching regular expression
awk '$3 == 65' testfile - print only lines with a third field value of 65
awk '$5 < = 3000' testfile - print only lines with a fifth field value that is less than or equal to 3000

awk '{print $1}' testfile - print first field of every record
awk '{print $3 $1}' testfile
awk '{print $3, $1}' testfile - inserts output field separator (variable OFS, default is space)

awk -F, '{print $2}' testfile - specifies that , is input field separator, default is space or tab
awk '$2 ~ /[0-9]/ {print $3, $1}' testfile - searches for reg-exp (a digit) only in the second field

awk '{printf "%-30s%20s\n", $3, $2}' testfile - print 3rd field left-justified in a 30 character field, 2nd field right-justified in a 20 character field, then skip to a new line (required with printf)

awk '$3 <= 23' testfile - prints lines where 3rd field has a value <= 23
awk '$3 <='$var1' {print $3}' testfile - $var1 is a shell variable, not an awk variable, e.g. first execute: var1=23

awk '$3<='$2' {$3++} {print $0}' testfile - if field 3 <= argument 2 then increment field 3, e.g. first execute: set xxx 23

awk '$3> 1 && $3 < 23' testfile - prints lines where 3rd field is in range 1 to 23

awk '$3 < 2 || $3 > 4' testfile - prints lines where 3rd field is outside of range 2 to 4
awk '$3 < "4"' testfile - double quotes force string comparison
NF is an awk variable meaning # of fields in current record

awk '! (NF == 4)' testfile - lines without 4 fields
NR is an awk variable meaning # of current record
awk 'NR == 2,NR==7' testfile - range of records from record number 2 to 7

BEGIN is an awk pattern meaning "before first record processed"
awk 'BEGIN {OFS="~"} {print $1, $2}' testfile - print 1st and 2nd field of each record, separated by ~
END is an awk pattern meaning "after last record processed"

awk '{var+=$3} END {print var}' testfile - sum of 3rd fields in all records
awk '{var+=$3} END {print var/NR}' testfile - average of 3rd fields in all records - note that awk handles decimal arithmetic

awk '$5 > var {var=$5} END {print var}' testfile - maximum of 5th fields in all records
awk '$5 > var {var=$5} END {print var}' testfile - maximum of 5th fields in all records

sort -rk5 testfile | awk 'NR==1 {var=$5} var==$5 {print $0}' - print all records with maximum 5th field

Simple awk operations involving functions within the command line:

awk '/chevy/' testfile

# Match lines (records) that contain the keyword chevy note that chevy is a regular expression...

awk '{print $3, $1}' testfile

# Pattern not specified - therefore, all lines (records) for fields 3 and 1 are displayed

# Note that comma (,) between fields represents delimiter (ie. space)

awk '/chevy/ {print $3, $1}' testfile

# Similar to above, but for chevy

awk '/^h/' testfile

# Match testfile that begin with h

awk '$1 ~ /^h/' testfile ### useful ###

# Match with field #1 that begins with h

awk '$1 ~ /h/' testfile

# Match with field #1 any epression containing the letter h

awk '$2 ~ /^[tm]/ {print $3, $2, "$" $5}' testfile

# Match testfile that begin with t or m and display field 3 (year), field 2 (model name) and then $ followed by field 4 (price)

--------------------------------------------------------------------------------------------------

Complex awk operations involving functions within the command line:

awk ?/chevy/ {print $3, $1}? testfile

# prints 3rd & 1st fields of record containing chevy

awk ?$1 ~ /^c/ {print $2, $3}? testfile

# print 2nd & 3rd fields of record with 1st field beginning with c

awk ?NR==2 {print $1, $4}? testfile

# prints 1st & 4th fields of record for record #2

awk ?NR==2, NR==8 {print $2, $3}? testfile

# prints 2nd & 3rd fields of record for records 2 through 8

awk ?$3 >= 65 {print $3, $1}? testfile

# prints 3rd & 1st fields of record with 3rd field >= 65

awk ?$5 >= ?2000? && $5 < ?9000? {print $2, $3}? testfile

# prints 2nd & 3rd fields of record within range of 2000 to under 9000

MySQL Replication : Purged binary logs

2009-07-28T17:57:00.000-07:00

Yesterday I have found that there is no space left on server of MySQL master and on Slave. Once I debugged I have come to know that there is GB's of bin-log files on Master and relay-log on Slave.

Its due to I have forget to add a expire_logs_days Variable in my.cnf during the configuration of replication server.

# expire_logs_days = 7

It will purged binary logs older than 7 days.The old logs will be purged during the next bin-log switch.

Or, You can also delete bin-log manually using command :

PURGE BINARY LOGS TO 'mysql-bin.010';
PURGE BINARY LOGS BEFORE '2008-04-02 22:46:26';

MYSQL: slow queries log

2009-07-02T09:36:00.000-07:00

MySQL has built-in functionality that allows you to log SQL queries to a file , You can enable the full SQL queries logs to a file or only slow running queries log. It is easy for us to troubleshoot/ debug the sql statement if SQL queries log enable , The slow query log is used to find queries that take a long time to execute and are therefore candidates for optimization.

To enable you just need to add some lines to your my.cnf file, and restart. Add the following:

* To enable slow Query Log only

log-slow-queries = /var/log/mysql/mysql-slow.log
long_query_time = 1

After enabling slow query, mysqld writes a statement to the slow query log file and it consists of all SQL statements that took more than long_query_time seconds to execute. The time to acquire the initial table locks is not counted as execution time. mysqld only log after SQL statements has been executed and after all locks have been released, so log order might be different from execution order. The minimum and default values of long_query_time are 1 and 10, respectively.

* To enable full Log Query

log=/var/log/mysqldquery.log

The above will log all queries to the log file.

Selecting Queries to Optmize
• The slow query log
– Logs all queries that take longer than long_query_time
– Can also log all querie s that don’t use indexes with
--log-queries-not-using-indexes
– To log slow administatve commands use
--log-slow-admin-statements
– To analyze the contents of the slow log use
mysqldumpslow

MYSQL: Query Execution Basics

2009-06-30T11:26:00.000-07:00

1. The client sends the SQL statement to the server.
-> The protocol is halfduplex, which means that at any given time the MySQL server can be either sending or receiving messages, but not both. It also means there is no way to cut a message short.
The client sends a query to the server as a single packet of data. This is why the max_packet_size configuration variable is important if you have large queries. Once the client sends the query, it doesn’t have the ball anymore; it can only wait for results. The response from the server usually consists of many packets of data.When the server responds, the client has to receive the entire result set. It cannot simply fetch a few rows and then ask the server not to bother sending the rest. If the client needs only the first few rows that are returned, it either has to wait for all of the server’s packets to arrive and then discard the ones it doesn’t need, or disconnect ungracefully. Neither is a good idea, which is why appropriate LIMIT clauses are so important.Here’s another way to think about this: when a client fetches rows from the server, it thinks it’s pulling them. But the truth is, the MySQL server is pushing the rows as it generates them. The client is only receiving the pushed rows; there is no way for it to tell the server to stop sending rows. The client is “drinking from the fire hose,” so to speak.

2. The server checks the query cache. If there’s a hit, it returns the stored result from the cache; otherwise, it passes the SQL statement to the next step.
-> Before even parsing a query, MySQL checks for it in the query cache, if the cache is enabled. This operation is a case sensitive hash lookup. If the query differs from a similar query in the cache by even a single byte, it won’t match, and the query processing will go to the next stage.
If MySQL does find a match in the query cache, it must check privileges before
returning the cached query. This is possible without parsing the query, because
MySQL stores table information with the cached query. If the privileges are OK,
MySQL retrieves the stored result from the query cache and sends it to the client,
bypassing every other stage in query execution. The query is never parsed, optimized,
or executed.

3. The server parses, preprocesses, and optimizes the SQL into a query execution
plan.
-> MySQL’s parser breaks the query into tokens and builds a “parse tree”
from them. The parser uses MySQL’s SQL grammar to interpret and validate the
query. For instance, it ensures that the tokens in the query are valid and in the proper order, and it checks for mistakes such as quoted strings that aren’t terminated. The preprocessor then checks the resulting parse tree for additional semantics that the parser can’t resolve. For example, it checks that tables and columns exist, and it resolves names and aliases to ensure that column references aren’t ambiguous.Next, the preprocessor checks privileges. This is normally very fast unless your server has large numbers of privileges.
-> The parse tree is now valid and ready for the optimizer to turn it into a query execution plan. A query can often be executed many different ways and produce the same result. The optimizer’s job is to find the best option. MySQL uses a cost-based optimizer, which means it tries to predict the cost of various execution plans and choose the least expensive. The unit of cost is a single random four-kilobyte data page read.

4. The query execution engine executes the plan by making calls to the storage engine API.
-> The parsing and optimizing stage outputs a query execution plan, which MySQL’s
query execution engine uses to process the query. The plan is a data structure; it is
not executable byte-code, which is how many other databases execute queries. In contrast to the optimization stage, the execution stage is usually not all that complex: MySQL simply follows the instructions given in the query execution plan.
Many of the operations in the plan invoke methods implemented by the storage
engine interface, also known as the handler API. Each table in the query is represented by an instance of a handler. If a table appears three times in the query, for example, the server creates three handler instances. Though we glossed over this before, MySQL actually creates the handler instances early in the optimization stage. The optimizer uses them to get information about the tables, such as their column names and index statistics.

5. The server sends the result to the client.
-> The final step in executing a query is to reply to the client. Even queries that don’t return a result set still reply to the client connection with information about the query, such as how many rows it affected.

How do I Use the Linux Top Command?

2009-06-14T03:21:00.000-07:00

The Unix top command is designed to help users determine which processes are running and which applications are using more memory or processing power than they should be.

The top command is very easy to use but you should know the things in details. The output of to is :

top output:

top - 22:09:08 up 14 min, 1 user, load average: 0.21, 0.23, 0.30
Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.5%us, 31.2%sy, 0.0%ni, 27.0%id, 7.6%wa, 1.0%hi, 23.7%si, 0.0%st
Mem: 255592k total, 167568k used, 88024k free, 25068k buffers
Swap: 524280k total, 0k used, 524280k free, 85724k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3166 apache 15 0 29444 6112 1524 S 6.6 2.4 0:00.79 httpd
3161 apache 15 0 29444 6112 1524 S 5.9 2.4 0:00.79 httpd
3164 apache 15 0 29444 6112 1524 S 5.9 2.4 0:00.75 httpd
3169 apache 15 0 29444 6112 1524 S 5.9 2.4 0:00.74 httpd
3163 apache 15 0 29444 6112 1524 S 5.6 2.4 0:00.76 httpd
3165 apache 15 0 29444 6112 1524 S 5.6 2.4 0:00.77 httpd
3167 apache 15 0 29444 6112 1524 S 5.3 2.4 0:00.73 httpd
3162 apache 15 0 29444 6112 1524 S 5.0 2.4 0:00.77 httpd
3407 root 16 0 2188 1012 816 R 1.7 0.4 0:00.51 top
240 root 15 0 0 0 0 S 0.3 0.0 0:00.08 pdflush
501 root 10 -5 0 0 0 S 0.3 0.0 0:01.20 kjournald
2794 root 18 0 12720 1268 560 S 0.3 0.5 0:00.73 pcscd
1 root 15 0 2060 636 544 S 0.0 0.2 0:03.81 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.07 events/0

The first line in top:

top - 22:09:08 up 14 min, 1 user, load average: 0.21, 0.23, 0.30

“22:09:08″ is the current time; “up 14 min” shows how long the system has been up for; “1 user” how many users are logged in; “load average: 0.21, 0.23, 0.30″ the load average of the system (1minute, 5 minutes, 15 minutes).

Load average is an extensive topic and to understand its inner workings can be daunting. The simplest of definitions states that load average is the cpu utilization over a period of time. A load average of 1 means your cpu is being fully utilized and processes are not having to wait to use a CPU. A load average above 1 indicates that processes need to wait and your system will be less responsive. If your load average is consistently above 3 and your system is running slow you may want to upgrade to more CPU’s or a faster CPU.

The second line in top:

Tasks: 82 total, 1 running, 81 sleeping, 0 stopped, 0 zombie

Shows the number of processes and their current state.

The third lin in top:

Cpu(s): 9.5%us, 31.2%sy, 0.0%ni, 27.0%id, 7.6%wa, 1.0%hi, 23.7%si, 0.0%st

Shows CPU utilization details. “9.5%us” user processes are using 9.5%; “31.2%sy” system processes are using 31.2%; “27.0%id” percentage of available cpu; “7.6%wa” time CPU is waiting for IO.

When first analyzing the Cpu(s) line in top look at the %id to see how much cpu is available. If %id is low then focus on %us, %sy, and %wa to determine what is using the CPU.

The fourth and fifth lines in top:

Mem: 255592k total, 167568k used, 88024k free, 25068k buffers
Swap: 524280k total, 0k used, 524280k free, 85724k cached

Describes the memory usage. These numbers can be misleading. “255592k total” is total memory in the system; “167568K used” is the part of the RAM that currently contains information; “88024k free” is the part of RAM that contains no information; “25068K buffers and 85724k cached” is the buffered and cached data for IO.

So what is the actual amount of free RAM available for programs to use ?

The answer is: free + (buffers + cached)

88024k + (25068k + 85724k) = 198816k

How much RAM is being used by progams ?

The answer is: used - (buffers + cached)

167568k - (25068k + 85724k) = 56776k

The processes information:

Top will display the process using the most CPU usage in descending order. Lets describe each column that represents a process.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3166 apache 15 0 29444 6112 1524 S 6.6 2.4 0:00.79 httpd

PID - process ID of the process

USER - User who is running the process

PR - The priority of the process

NI - Nice value of the process (higher value indicates lower priority)

VIRT - The total amount of virtual memory used

RES - Resident task size

SHR - Amount of shared memory used

S - State of the task. Values are S (sleeping), D (uninterruptible sleep), R (running), Z (zombies), or T (stopped or traced)

%CPU - Percentage of CPU used

%MEM - Percentage of Memory used

TIME+ - Total CPU time used

COMMAND - Command issued
Interacting with TOP

Now that we are able to understand the output from TOP lets learn how to change the way the output is displayed.

Just press the following key while running top and the output will be sorted in real time.

M - Sort by memory usage

P - Sort by CPU usage

T - Sort by cumulative time

z - Color display

k - Kill a process

q - quit

If we want to kill the process with PID 3161, then press “k” and a prompt will ask you for the PID number, and enter 3161.
Command Line Parameters with TOP

You can control what top displays by issuing parameters when you run top.

- d - Controls the delay between refreshes

- p - Specify the process by PID that you want to monitor

-n - Update the display this number of times and then exit

If we want to only monitor the http process with a PID of 3166

$ top -p 3166

If we want to change the delay between refreshes to 5 seconds

$ top -d 5

Block IP addresses using IPtables

2009-04-17T23:00:00.000-07:00

Block a particular
#service iptables start
#iptables -I INPUT -s 10.1.24.4 -j DROP

This command will simply drop any packet coming from the address 10.1.24.4

To list the chains:
#iptables -L -n

To make persist :

#service iptables status
#iptables-save (copy output)
#emacs /etc/sysconfig/iptables (paste output)
#service iptables restart

make sure iptables service start on default.

creating bulk users in linux

2009-03-30T06:16:00.000-07:00

Today I have configured NX server.
Now, Next task is to create users and really its very time consuming and boring task.

Usually you use useradd command to create a new user or update default new user information from command line.

So i have explore Linux and searched on Google , I have found few scripts to do this. But later I have found one good and easy solution.

Here is that,

Update and create new users in bulk.

newusers command reads a file of user name and clear-text password pairs and uses this information to update a group of existing users or to create new users. Each line is in the same format as the standard password file.

This command is intended to be used in a large system environment where many accounts are updated at a single time (batch mode). Since username and passwords are stored in clear text format make sure only root can read/write the file. Use chmod command:
# touch /root/bulk-user-add.txt
# chmod 0600 /root/bulk-user-add.txt

Create a user list as follows. Open file:
# emacs /root/bulk-user-add.txt

Append username and password:
sanjay:mypass99:555:555:Sanjay Singh:/home/Sanjay:/bin/bash
frampton:mypass99n:556:556:Frampton Martin:/home/Frampton:/bin/bash
----
--
---
barun:mypass99:560:560:Barun Ghosh:/home/Barun:/bin/bash

Now create users in batch:
# newusers /root/bulk-user-add.txt

Read man page of newusers for more information.
May be I will automate entire procedure using a php

Boot time parameters of Linux kernel

2009-03-24T11:04:00.000-07:00

Boot time parameters you should know about the Linux kernel.

The Linux kernel accepts boot time parameters as it starts to boot system. This is used to inform kernel about various hardware parameter.

The kernel command line syntax
name=value1,value2,value3…
Where,
* name : Keyword name, for example, init, ro, boot etc

Common Boot time parameters
init
This sets the initial command to be executed by the kernel. Default is to use /sbin/init, which is the parent of all processes.
To boot system without password pass /bin/bash or /bin/sh as argument to init
init=/bin/bash

single
The most common argument that is passed to the init process is the word 'single' which instructs init to boot the computer in single user mode, and not launch all the usual daemons

root=/dev/device
This argument tells the kernel what device (hard disk, floppy disk) to be used as the root filesystem while booting. For example following boot parameter use /dev/sda1 as the root file system:
root=/dev/sda1
If you copy entire partition from /dev/sda1 to /dev/sdb1 then use
root=/dev/sdb1

ro
This argument tells the kernel to mount root file system as read-only. This is done so that fsck program can check and repair a Linux file system. Please note that you should never ever run fsck on read/write file system.

rw
This argument tells the kernel to mount root file system as read and write mode.

panic=SECOND
Specify kernel behavior on panic. By default, the kernel will not reboot after a panic, but this option will cause a kernel reboot after N seconds. For example following boot parameter will force to reboot Linux after 10 seconds
panic=10

maxcpus=NUMBER
Specify maximum number of processors that an SMP kernel should make use of. For example if you have four cpus and would like to use 2 CPU then pass 2 as a number to maxcpus (useful to test different software performances and configurations).
maxcpus=2

debug
Enable kernel debugging. This option is useful for kernel hackers and developers who wish to troubleshoot problem

selinux [0|1]
Disable or enable SELinux at boot time.
• Value 0 : Disable selinux
• Value 1 : Enable selinux

raid=/dev/mdN
This argument tells kernel howto assembly of RAID arrays at boot time. Please note that When md is compiled into the kernel (not as module), partitions of type 0xfd are scanned and automatically assembled into RAID arrays. This autodetection may be suppressed with the kernel parameter "raid=noautodetect". As of kernel 2.6.9, only drives with a type 0 superblock can be autodetected and run at boot time.

mem=MEMEORY_SIZE
This is a classic parameter. Force usage of a specific amount of memory to be used when the kernel is not able to see the whole system memory or for test. For example:
mem=1024M
The kernel command line is a null-terminated string currently up to 255 characters long, plus the final null. A string that is too long will be automatically truncated by the kernel, a boot loader may allow a longer command line to be passed to permit future kernels to extend this limit (H. Peter Anvin ).
Other parameters
initrd /boot/initrd.img
An initrd should be loaded. the boot process will load the kernel and an initial ramdisk; then the kernel converts initrd into a "normal" ramdisk, which is mounted read-write as root device; then /linuxrc is executed; afterwards the "real" root file system is mounted, and the initrd file system is moved over to /initrd; finally the usual boot sequence (e.g. invocation of /sbin/init) is performed. initrd is used to provide/load additional modules (device driver). For example, SCSI or RAID device driver loaded using initrd.

hdX =noprobe
Do not probe for hdX drive. For example, disable hdb hard disk:
hdb=noprobe
If you disable hdb in BIOS, Linux will still detect it. This is the only way to disable hdb.

ether=irq,iobase,[ARG1,ARG2],name
Where,
• ether: ETHERNET DEVICES
For example, following boot argument force probing for a second Ethernet card (NIC), as the default is to only probe for one (irq=0,iobase=0 means automatically detect them).
ether=0,0,eth1

How to begin the enter parameters mode?
You need to enter all this parameter at Grub or Lilo boot prompt. For example if you are using Grub as a boot loader, at Grub prompt press 'e' to edit command before booting.
1) Select second line
2) Again, press 'e' to edit selected command
3) Type any of above parameters.
See an example of "recovering grub boot loader password", for more information. Another option is to type above parameters in grub.conf or lilo.conf file itself.

More detail

MYSQL REPLICATION & DISASTER RECOVERY

2009-03-22T00:18:00.000-07:00

MySQL’s built-in replication capability is the foundation for building large, high performance applications on top of MySQL. Replication lets you configure one or more servers as slaves, or replicas, of another server.

1. Setting up Replication:

Three threads are involved in Replication: One on the master and two on the slave.
• The I/O thread on the slave connects to the master and requests the binary update log. The Binary log dump thread on the master sends the binary update log to the slave on request.
• Once on the slave, the I/O thread reads the data sent by the master and copies it to the relay log in the data directory.
• The third thread, also on the slave, is the SQL Thread, which read and executes the queries from the relay log to bring the slave in alignment with the master.

2. Replication with example:

Version: Both master and slave should be the same version. Otherwise replication will be improper.
Network Configuration Settings
Master Server IP Address: 10.5.1.10
Slave Server Primary IP Address (Ethernet eth0): 10.5.1.11
Slave Server Secondary IP Address (Ethernet eth1): 10.5.1.10 (By default it is disabled)

3. MySQL Replication Installation:

Step1:

Install MySQL on master 1 and slave 1. Configure network services on both systems, like
Master 1/Slave 2 IP: 10.5.1.10
Master 2/Slave 1 IP: 10.5.1.11

Step2:

On Master 1, make changes in my.cnf:
[mysqld]
logbin= mysqlbin
binlogdodb= # input the database which should be replicated or ignore this command to replicate all the databases.
binlogignoredb= mysql # input the database that should be ignored for replication
serverid=1
auto_increment_increment=2
auto_increment_offset=1

Step 3:

On master 1, create a replication slave account in mysql.
mysql> grant replication slave on *.* to 'replication'@10.5.1.11 identified by 'slave';
Restart the mysql master1.

Step 4:

Now edit my.cnf on Slave1 or Master2:
[mysqld]
serverid =2
masterhost= 10.5.1.10
masteruser= replication
masterpassword= slave
masterport= 3306

Step 5:

Restart th MySQL Slave 1.
Login to the MySQL command prompt and start the slave replication.
[root@Slavetest ~]# mysql -u root -p
Enter password: xxxxx (Please consult MySQL Administrator/IT Manager)
Welcome to the MySQL monitor. Commands end with; or \g.
Your MySQL connection id is 250
Server version: 5.0.54a-enterprise-gpl-log MySQL Enterprise Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
mysql> start slave;
mysql> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.5.1.10
Master_User: replication
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000018
Read_Master_Log_Pos: 2953
Relay_Log_File: slavetest-relay-bin.000065
Relay_Log_Pos: 235
Relay_Master_Log_File: mysql-bin.000018
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 2953
Relay_Log_Space: 235
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
1 row in set (0.00 sec)
mysql>
Above highlighted rows must be indicate related log files and Slave_IO_Running and
Slave_SQL_Running: must be to YES.

Step 6:

On Master 1:
[root@TESTDB~]# mysql -u root -p
Enter password: xxxxx (Please consult MySQL Administrator/IT Manager)
Welcome to the MySQL monitor. Commands end with; or \g.
Your MySQL connection id is 250
Server version: 5.0.54a-enterprise-gpl-log MySQL Enterprise Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
mysql> show master status;
+------------------------------------+-----------+--------------------+------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB|
+------------------------------------+-----------+--------------------+------------------------+
|MysqlMYSQL01-bin.000008| 410 | | mysql |
+------------------------------------+-----------+--------------------+------------------------+
1 row in set (0.00 sec)
The above scenario is for master-slave, now we will create a slave master scenario for the same systems and it will work as master master.

Step 7:

Edit on Master 2/ Slave 1, edit my.cnf and master entries into it:
[mysqld]
logbin=mysqlbin #information for becoming master added
binlogignoredb=mysql

Step 8:

Create a replication slave account on master2 for master1:
mysql> grant replication slave on *.* to 'slavereplication'@10.5.1.10 identified by 'slave';

Step 9:

Edit my.cnf on master1 for information of its master.
[mysqld]
auto_increment_increment=2
auto_increment_offset=1 #information for becoming slave.
masterhost= 10.5.1.11
masteruser= slavereplication
masterpassword= slave
masterport= 3306
master_connect_retry=60

Step 10:

Restart both mysql master1 and master2.

Step 11:

Monitor Mysql Replication using :
• show slave status\G
• show processlist\G
• show master status;

4. Fail over Configuration Procedure :

Step1:

Login to Slave Server Using Secure Shell in Linux or Putty tool
[root@TESTDB ~]# ssh 10.5.1.11
root@10.5.1.11 password: xxxxx
Last login: Tue Oct 7 15:30:41 2008 from 10.5.1.10
[root@slavetest ~]#
or Use Putty Tool in Windows
Enter the slave IP address and user name and password
After login into slave machine proceed the below steps.

Step2:

Connect the Ethernet Interface cable.

Step3:

Enable the secondary Ethernet interface
[root@slavetest ~]# ifup eth1
Step4: Login into Mysql
[root@slavetest ~]# mysql -u root -p
Enter password: xxxxx (Please consult MySQL Administrator/IT Manager)
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 250
Server version: 5.0.54a-enterprise-gpl-log MySQL Enterprise Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
You Will get Welcome message and mysql prompt as show above.

Step5:

Check the Slave Status
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.5.1.10
Master_User: replication
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000018
Read_Master_Log_Pos: 2953
Relay_Log_File: slavetest-relay-bin.000065
Relay_Log_Pos: 235
Relay_Master_Log_File: mysql-bin.000018
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 2953
Relay_Log_Space: 235
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
1 row in set (0.00 sec)
mysql>
Above Result will display Slave Replication status.
Make sure that the slave has processed any statements in their relay log. On slave, issue STOP SLAVE IO_THREAD, then check the output of SHOW PROCESSLIST until you see Has read all relay log; waiting for the slave I/O thread to update it. When this is true for all slaves, they can be reconfigured to the new setup.
mysql> Stop Slave io_thread;
mysql> show processlist\G
*************************** 1. row ***************************
Id : 203
User: slavereplication
Host: 10.5.1.10:59795
db : NULL
Command: Binlog Dump
Time: 158086
State: Has sent all binlog to slave; waiting for binlog to be updated
Info: NULL
*************************** 2. row ***************************
Id: 230
User: system user
Host:
db: NULL
Command: Connect
Time: 152220
State: Has read all relay log; waiting for the slave I/O thread to update it
Info: NULL
*************************** 3. row ***************************
Id: 251
User: root
Host: localhost
db: NULL
Command: Query
Time: 0
State: NULL
Info: show processlist
3 rows in set (0.00 sec)
mysql>

Step6:

Making Slave as Master server
mysql> show master status;
+----------------------+----------+-------------------+-----------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+----------------------+----------+-------------------+-----------------------------+
| mysql-bin.000006 | 783 | | mysql,mysql |
+----------------------+----------+-------------------+-----------------------------+
1 row in set (0.00 sec)
mysql>
(Note : This is very important make a note down of mysql log file and position when the slave becomes master).
mysql> stop slave;
This command will stop the slave replication. Now it is ready to serve as a master, start the application in the server.

5. Restoration Procedure:

Note: Bring down the secondary Ethernet interface eth1 down in slave server before your plan to restore. Now once the original master server problem has been fixed and making it as live server. When Master is up again, you must issue the CHANGE MASTER, so that Master becomes a slave of S1 and picks up each Web Client writes that it missed while it was down.

Step1:

Connect to the Current Master server
[root@TESTDB ~]# ssh 10.5.1.11
root@10.5.1.11's password: xxxxxxx
Last login: Tue Oct 7 15:30:41 2008 from 10.5.1.10
[root@mastertest ~]#
or
Use Putty Tool in Windows
Enter the slave IP address and user name and password
After login into machine proceed the below steps.

Step 2:

Unplug the secondary Ethernet Interface in current Master.

Step3:

Disable the secondary Ethernet Interface in current Master.
[root@mastertest ~]# ifdown eth1
Now Boot the Original Master Server and if the Ethernet cable is UN-plugged. Plug the Ethernet Interface in the Original Master server.

Step4:

Login To Original_Mater Server Using Secure Shell in Linux or Putty tool
[root@TESTDB ~]# ssh 10.5.1.10
root@10.5.1.10's password: xxxxxxx
Last login: Tue Oct 7 15:30:41 2008 from 10.5.1.10
[root@mastertest ~]#
Or Use Putty Tool in Windows
Enter the slave IP address and user name and password
After login into machine proceed the below steps.

Step5:

Login into Mysql
[root@slavetest ~]# mysql -u root -p
Enter password: xxxxx (Please consult MySQL Administrator/IT Manager)
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 250
Server version: 5.0.54a-enterprise-gpl-log MySQL Enterprise Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
You Will get Welcome message and mysql prompt as show above.

Step6:

Configure Old_master as a Slave Server of Current_Master Server
mysql>change master to
>master_host=10.5.1.11', (slave server ip address)
>master_user='slavereplication', (slavereplication-mysql replication user created in the
slave server)
>master_password='slave', (slave- mysql replication user password created in the
slave server)
>master_log_file='mysql-bin.0000xx', (xx: position of current_master server log file name as shown in show master status, at the time of
making slave as master)
> master_log_pos=xx; (xx:position of current_master log position as shown
in show master status, at the time of making slave as master)
mysql>
mysql>start slave;
Now it will get updates, which are missed during the failure. Once all update has been finished make master a master again.

Step7:

To make Master a master again (because it is the most powerful machine, for example), use the preceding procedure as if Slave 1 was unavailable and Master was to be the new master.
[root@mastertest ~]# service mysql stop
[root@mastertest ~]# cd /var/lib/mysql
[root@mastertest ~]# mv master.info relay-log.info servername-relay.bin* /root
[root@mastertest ~]# service mysql start

Step8:

Making original Slave as a slave replication server

[root@slavetest ~]# mysql -u root -p
Enter password: xxxxx (Please consult MySQL Administrator/IT Manager)
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 250
Server version: 5.0.54a-enterprise-gpl-log MySQL Enterprise Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> start slave;

6. Replication Files:

Let’s take a look at some of the files replication uses. You already know about the binary log and the relay log, but there are several other files too. Where MySQL places them depends mostly on your configuration settings. Different MySQL versions place them in different directories by default. You can probably find them either in the data directory or in the directory that contains the server’s .pid file (possibly /var/run/mysqld/ on Unix-like systems).
Here they are:
• mysql-bin.index : A server that has binary logging enabled will also have a file named the same as the binary logs, but with a .index suffix. This file keeps track of the binary log files that exist on disk. It is not an index in the sense of a table’s index; rather, each line in the file contains the filename of a binary log file. You might be tempted to think that this file is redundant and can be deleted (after all, MySQL could just look at the disk to find its files), but don’t. MySQL relies on this index file, and it will not recognize a binary log file unless it’s mentioned here.

• mysql-relay-bin.index : This file serves the same purpose for the relay logs as the binary log index file does for the binary logs.

• master.info : This file contains the information a slave server needs to connect to its master. Don’t delete it, or your slave will not know how to connect to its master after it restarts. This file contains the replication user’s password, in plain text, so you may want to restrict its permissions.

• relay-log.info: This file contains the slave’s current binary log and relay log coordinates (i.e., the slave’s position on the master). Don’t delete this either, or the slave will forget where it was replicating from after a restart and might try to replay statements it has already executed.

These files are a rather crude way of recording MySQL’s replication and logging state. Unfortunately, they are not written synchronously, so if your server loses power and the files haven’t yet been flushed to disk, they can be inaccurate when the server restarts.

move you bookmarks between computers

2008-11-20T01:57:00.001-08:00

Most of time I face problem when I lose my bookmarks. Its important to take backup of bookmarks which easily transfer between systems. After doning 10min of search I comeup with 3 method.
a. Export & Import Method.

From the Firefox browser, click on on the "Bookmarks" menu located at the top of the window. Next, click on the "Organize Bookmarks"
This brings you to a separate window which lists all existing bookmarks (favorite websites). From the File menu at the top, you have the option to Import and Export.
If you need to import external bookmarks and merge them into Firefox, do so by clicking "Import" from the File menu. This allows you to import your favorites from Internet Explorer, or just an existing file you had saved in the past. Choose the desired option, and hit "Next" to search for your bookmarks. When done, click "Finish."
If you need to externally save existing Firefox bookmarks, click on the "Export" option from the File menu. Once you choose a save destination, you're all set.

b. Keep your bookmark online, few sites gives service to keep the bookmarks online and it will sync data between your firefok and there server. Just you need to create account there one of such site is http://www.mybookmarks.com/.

c. This will take backup of all your Firefox Extentions along with bookmarks. What you need to do install FEBE (Firefox Environment Backup Extension). There is another extension CLEO(Compact Library Extension Organizer) that can be used to combine all the extension and make it as a single extension file. it can also be very help full if you have your own custom made extension.

source :

http://mozilla.gunnars.net/firefox_bookmarks_tutorial.html

http://prabhanjan-panigrahi.blogspot.com/2008/11/backup-your-firefox-extentions-and.html

Performance : Alternative PHP Cache (APC)

2008-09-25T05:32:00.000-07:00

1. yum install php-pecl-apc

If for some reason centosplus repo is not enabled then:
1. cd /usr/src/
2. wget http://pecl.php.net/get/APC-3.0.19.tgz
3. tar xvzf APC-3.0.19.tgz
4. cd APC-3.0.19
5. yum install php-devel
6. yum install automake
7. yum install libtool httpd-devel*
8. /usr/bin/phpize
9. ./configure --enable-apc --enable-apc-mmap --with-apxs --with-php-config=/usr/bin/php-config
10. make
11. make install
12. ls /usr/lib64/php4/ or /usr/lib64/php/modules/
13 #emacs /etc/php.ini
extension=apc.so
apc.enabled=1
apc.shm_segments=1
apc.shm_size=128
apc.ttl=7200
apc.user_ttl=7200
apc.num_files_hint=1024
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.enable_cli=1

Understanding LVM

2008-09-15T06:25:00.000-07:00

LVM terminology:
a. Physical Volume (PV) : A PV is nothing more than a physical medium with some administrative data added to it - once you have added this, LVM will recognise it .
b. Physical Extents (PE) :- Physical Extents are like really big blocks, often with a size of megabytes.
c. Volume Group (VG) - A VG is made up of a number of Physical Extents (which may have come from multiple Physical Volumes or hard drives). While it may be tempting to think of a VG as being made up of several hard drives (/dev/hda and /dev/sda for example), it's more accurate to say that it contains PEs which are provided by these hard drives.
d. Logical Volumes (LV) - A logical volume functions like a normal partition -- it have a filesystem such as Ext3, and a mount point. From , Volume Group, PEs can be assigned to a logical Volume (LV).

Creating Physical volumes:
#pvcreate /dev/sda3

Multiple physical volumes
#pvcreate /dev/sda9 /dev/sda10

Creating a volume group:
#vgcreate home2 /dev/sda3

Multiple physical volumes to make up a volume group
#vgcreate home3 /dev/sda9 /dev/sda10

Creating logical volume:
#lvcreate --size 1G -n downloads home2
now,
#mkfs.ext3 /dev/home2/downloads
#mkdir /home/downloads
#mount -t ext3 /dev/home2/downloads /home/downloads
Add a line to the /etc/fstab file
/dev/home2/downloads /home/downloads ext3 defaults 1 2

Resizing logical volumes:
#umount /home/downloads
#lvextend -L +1G /dev/home2/downloads
#resize2fs /dev/home2/downloads

Similar to the lvextend there is the lvreduce command:
#lvreduce -L -500M /dev/home2/downloads
#resize2fs /dev/home2/downloads

Modifying volume groups:
a. unmount the logical volumes within your volume group.
b. #vgextend home2 /dev/sda5 /dev/sda7
(The volume group home2 is now made up of sda3, sda5, and sda7)

Remove a volume group
a. Unmount your logical volume
b. #vgremove home2

A Physical Volume, containing Physical Extents:

+-----[ Physical Volume ]------+
| PE | PE | PE | PE | PE | PE  |
+------------------------------+

A Volume Group, containing 2 Physical Volumes (PVs) with 6 Physical Extents:

+------[ Volume Group ]-----------------+
|  +--[PV]--------+  +--[PV]---------+  |
|  | PE | PE | PE |  | PE | PE | PE  |  |
|  +--------------+  +---------------+  |
+---------------------------------------+

We now further expand this:

+------[ Volume Group ]-----------------+
|  +--[PV]--------+  +--[PV]---------+  |
|  | PE | PE | PE |  | PE | PE | PE  |  |
|  +--+---+---+---+  +-+----+----+---+  |
|     |   |   | +-----/     |    |      |
|     |   |   | |           |    |      |
|   +-+---+---+-+      +----+----+--+   |
|   |  Logical  |      |  Logical   |   |
|   |  Volume   |      |   Volume   |   |
|   |           |      |            |   |
|   |  /home    |      |    /var    |   |
|   +-----------+      +------------+   |
+---------------------------------------+

You can use the lvdisplay command see logical vloume's status.

http://www.ibm.com/developerworks/library/l-lvm2/index.html
http://www.linux.com/feature/11864

Roaming Profile on samba with PDC controller, windows as client

2008-09-15T05:16:00.000-07:00

1. Make these change in samba configuration file : emacs /etc/samba/smb.conf
Important terms to understand
a. wins support = yes
b. domain master = yes
c. netbios name = samba
d. workgroup = OFFICE
e. domain logons = yes

2. Samba root access
#smbpasswd -a

3. Add new user :
#useradd prabhat (adding linux user)
#passwd prabhat
#smbpasswd -a prabhat (converting linux user into samba user)

4. On the windows machine,Aadd window machine in 'OFFICE' domain of samba.
On desktop, Right click on "My Computer" -> "properties" -> "Network identificatation" tab -> "change"
while joing it will ask for username password "root" & "OFFICE" (see serial no 2.)

Important Points:
1. There will be one and only one PDC over network.
2. The machine name must be unique over network.
3. While login from client domain name must be 'OFFICE'

#emacs /etc/samba/smb.conf

[global]
log file = /var/log/samba/%m.log
log level = 2
load printers = yes
idmap gid = 16777216-33554431
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
winbind use default domain = no
template shell = /bin/false
wins support = yes
dns proxy = no
netbios name = samba
domain master = yes
preferred master = yes
cups options = raw
server string = Samba PDC
idmap uid = 16777216-33554431
workgroup = OFFICE
os level = 64
domain logons = yes
add user script = /usr/sbin/useradd -m '%u'
delete user script = /usr/sbin/userdel -r '%u'
add group script = /usr/sbin/groupadd '%g'
delete group script = /usr/sbin/groupdel '%g'
add user to group script = /usr/sbin/usermod -G '%g' '%u'
set primary group script = /usr/sbin/usermod -g '%g' '%u'
add machine script = /usr/sbin/useradd -s /bin/false -d /dev/null '%u'
logon script = scripts\logon.bat
passdb backend = tdbsam
printcap name = /etc/printcap
security = user
max log size = 50
hosts allow = 10.1.

[homes]
comment = Home Directories
valid users = %S
browseable = No
writable = Yes

# Un-comment the following and create the netlogon directory for Domain Logons
[netlogon]
comment = Network Logon Service
logon path = \\%L\profiles\%U
logon home = \\%L\%U\.9xprofile
logon drive = p:
browseable = no
writable = no
; share modes = no

Concept:
http://searchenterpriselinux.techtarget.com/tip/0,289483,sid39_gci1151921,00.html
Practial Implement:
http://searchenterpriselinux.techtarget.com/tip/0,289483,sid39_gci1151926,00.html
Read section : "Joining" a Samba Domain
http://www.rmschneider.com/writing/xp_and_samba.html

NFS server Redhat 5

2008-07-23T05:24:00.000-07:00

Making NFS server Live:

Server side (Server IP 10.1.31.1):
1. Check nfs-utils is installed or not using
#yum list nfs-utils
2. Edit /etc/exports , add dir you want share with netrwork information and permision
# vi /etc/exports
/var/ftp/pub 10.1.0.0/16(rw,sync,no_root_squash)
3. Restart nfs service
# service nfs retstart
or
# service nfs reload
4. Check port & shared dir using
# rcpinfo -p
# export -v
# service portmap status

Client Side (Client IP : 10.1.0.0/16, eg 10.1.31.67):
1. mount shared nfs folder
# mount 10.1.31.3:/var/ftp/pub /mnt
2. Do static mount using /etc/fstab

# vi /etc/fstab
10.1.31.1:/var/ftp/pub /mnt nfs defaults 0 0

Creating RAID Partition

2008-07-16T04:17:00.000-07:00

Creating RAID-5

For this you need 3 Partition, 2 partitions with same size and 3rd partition is 10% greater than 1st or 2nd( since 1 & 2nd partition of same size.). You can use fdisk command to create partition and make sure partition type must be FD. In fdisk command use option t to change partition type.

eg. I have created /dev/sda1, /dev/sdb1 and /dev/sdc1.

Once you created 3 partitions use this to make RAID 5:
#mdadm --create /dev/md0 --level=5 --raid-device=3 /dev/sd{a1,b1,c1}

Check the RAID partition using:
#less /proc/mdstat

Or you can see detail information's using:
#mdadm --detail /dev/md0

To rebuild corrupted partation use:
#mdadm /dev/md0 -r /dev/sdb0

Swap Filesystem Creation

2008-07-10T04:39:00.000-07:00

Partition Creation for Swap filesystem
1. fdisk -l
2. fdisk /dev/sda
3. Type p & Press Enter
4. Type n & Press Enter
5. Just Press Enter for 1st Cylinder size
6. Type +sizeM for last Cylinder size
7. Type p & Press Enter
8. Type t & Press Enter
9. t2 /dev/sda12
10. Type 82 & Press Enter
11. Type p & Press Enter
12. Type w & Press Enter
13. partprobe
14. swapon -s
15. swapoff /dev/sda7 => Swapoff the another swap partition
16. swapon -s

Creation of a new Swap Filesystem
1. mkswap -v1 /dev/sda12
2. swapon /dev/sda12
3. swapon /dev/sda7
Here /dev/sda7 & /dev/sda12 are swap partition.

For automounting the either or swap partition do the following :

1. vi /etc/fstab
2. Add the following line :-

/dev/sda12 swap swap defaults 0 0

Note :- Partition Id identifies what type of filesystem will have to create. Id is Hexacode number. Maximum 4 partition can be create 3 primary & 1 Extended.

Filesystem ID
Ext2 / Ext3 ( Linux) 83
Swap 82
Fat32 C

Linux partition using fdisk

2008-07-10T03:48:00.000-07:00

Linux partition using fdisk

1. fdisk -l => To check what type of Hard Disk i.e. Sata ( /dev/sda ) or Pata ( /dev/hda )
2. fdisk /dev/sda
3. Type n then Enter => For new Partition
4. Just press Enter => For 1st Cylinder
5. Type +size and press Enter => For Last Cylinder for Example +1000M. Size can be in terms of M or K
6. Type p => To print the partition
7. Type w => To save the new partition. It's save in the hard disk but kernel partition table not updated. (/dev/sda11 is created in my system , might be different in your system)
8. Type partprobe & Press Enter => To update the kernel.
9. mkfs.ext3 -b 2048 /dev/sda11 => Here b is the block & is \block size in bytes & /dev/sda11 is new partition has created
11. e2label /dev/sda11 data=> data is Label name
12. e2label /dev/sda11
13. mkdir /data => To mount the partition /dev/sda11 /data is a directory
14. mount /dev/sda11 /data
15. ls /data
To make this above mount point for /dev/sda11 permanent write entry in the /etc/fstab file by the following way :-

1.vi /etc/fstab
2. LABEL=data /data ext3 defaults 0 0

=> defaults means how will it be mount by which criteria rw,exec,acl etc. First 0 is for dump value i.e. Whether there will be automatic dumping or not means or off. Next 0 is fsck sequence value i.e. If there is any improperly shutdown occurs then the system again then which file it will check for consistency.0 means it doesn't check.

instant Messenger: Openfire, Spark & web chat (Fastpath)

2008-05-02T18:47:00.000-07:00

-> Openfire
1 . lynx http://www.igniterealtime.org/downloads/download-landing.jsp?file=openfire/openfire-3.4.5-1.i386.rpm
2 . rpm -ivh openfire-3.4.5-1.i386.rpm
3 . /etc/rc.d/init.d/openfire status
4 . /etc/rc.d/init.d/openfire start
5. Make sure that the firewall is stopped.
6 . http://www.grmtech.com:9090/
7 . From the plugin tab in the web based admin control panel: installed Asterisk-IM Openfire

-> Spark
Download and install on client system.
http://www.igniterealtime.org/downloadServlet?filename=spark/spark_2_5_8.exe

-> web chat
Go to plugin tab in the web based admin control panel:
1.Click on 'Available Plugins'
2. Install the following plugin
a. Spark Fastpath Webchat
b. Fastpath Service
c. Monitoring Service (its is a optional)

How to make a linux server act as a gateway

2008-05-02T18:44:00.000-07:00

1. turned on IP forwarding in sysctl.conf by setting the value to 1
2. gave the ip tables command for nat masquerade.
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
Checked that the commands took effect by:
iptables-save
and then made the entries from iptables-save inside the file /etc/sysconfig/iptables

Check and Rebuilding failed Linux software RAID

2008-05-02T18:42:00.000-07:00

# cat /proc/mdstat
# mdadm -D /dev/md0
Detail : http://sitearticles.com/cms/show/43.html

ssh commands to log in without the password

2008-05-02T18:35:00.000-07:00

ssh commands to log in without the password
# ssh-keygen -t rsa
* This will generate your id_rsa and id_rsa.pub in the .ssh directory in your home directory
* copy the id_rsa.pub to the .ssh directory of the remote host you want to logon to as authorized_keys2
* If you have more than one host from which you want to connect to the remote host, you need to add the local host's id_rsa.pub as one line in the authorized_keys2 file of the remote host command for the 1and1 servers is:
# scp .ssh/id_rsa.pub u35894953@217.160.226.69:./.ssh/authorized_keys2

:)

Apache: Proxy and ReverseProxy Server

2008-04-13T23:30:00.000-07:00

Redirect your all traffic:
Case:
example.com A record is point to server with IP10.1.31.7 (mean its resolves to IP 10.1.31.7 ), but you are running your web server from example.com on server with IP 10.1.31.8.
So, you want all traffic coming to 10.1.31.7 will redirect to 10.1.31.8.

Do the following changes on server have IP 10.1.31.7 :
1. Run apache.
2. Do the following entry in apache.
a. NameVirtualHost 10.1.31.7
b <VirtualHost 10.1.31.7:80 >
ServerName example.com
ServerAlias www.example.com
DocumentRoot /home/test
ProxyRequests off
ProxyPass / http://www.example.com/
ProxyPassReverse / http://www.example.com/
<Location />
ProxyPassReverse /
</Location>
</VirtualHost>
3. Do the following entry in /etc/hosts.
10.1.31.8 example.com
10.1.31.8 www.example.com
(make sure once you #ping example.com from 10.1.31.7 it resolver's to 10.1.31.8).

Note: its redirect only http request not https.

more you can see on this link

Removing all blank lines, leading and trailing blank spaces

2008-04-12T06:55:00.000-07:00

Removing all blank lines, leading and trailing blank spaces :
#sed '/^$/d; s/^[ ]*//g; s/[ ]*$//g' prabhat.txt > kumar.txt

This command will create new file (kumar.txt) on which it removes blank lines, leading and trailing blank spaces.

Changing Run Levels

2008-04-11T00:29:00.000-07:00

On production environment normal run level 3 is used and it get rarely changed.
Runlevel and usage:

0 — Halt
1 — Single-user mode
2 — Not used (user-definable)
3 — Full multi-user mode
4 — Not used (user-definable)
5 — Full multi-user mode (with an X-based login screen)
6 — Reboot

Check run level using :-
#who -r or #runlevel

Change run level using:-
#telinit 5 or
open file
#emacs /etc/inittab
Look for the default runlevel called initdefault which look like as follows:
id:3:initdefault:
Replace run level x with y (where x is current run level and y you want to set).

Set services in run level using :
#ntsysv or
#chkconfig or#serviceconf

How To: Transfer your PuTTY settings between computers

2008-04-01T04:42:00.000-07:00

Putty stores its settings in the Windows registry. To save a backup of your Putty settings, you'll need to export this registry key to a file.

HKEY_CURRENT_USER\Software\SimonTatham

Steps :
1. Click Start->Run and type "RegEdt32" in the "Open" dialog. Click "Ok"
2. One RegEdt32 starts, you'll be presented with an application.
3. Press "Ctrl+F" to bring up the Find dialog. Enter the name of the key, "SimonTratham" in the "Find What" field, and make sure only "Keys" is checked in the "Look At" section of the dialog. Finally, click "Find Next"
4. The search may take a while, reminding us that the Windows Registry is a large and mysterious place where dragons be. Let's use these few seconds to reflect on the fact that you should never, ever, never change things in the registry unless you are absolutely, positively, totally, completely, 100% dead sure that you know exactly what you're doing. When the search completes we'll see the key name for which we're looking.
5. Click File->Export. Give your file an appropriate name like, "putty.reg" and click "Save"
6. We're done! Save the putty.reg file somewhere safe. The file doesn't contain any passwords or actual SSH key values so, it's relatively safe from prying eyes. Still, it does contain your configuration and that kind of data is a private matter.

Importing Your PuTTy Configuration:
Windows will ask you for confirmation that you want to import this set of registry values. We know this file is safe, because we created it but, you should never import registry information from an unknown source

Remove all empty directories

2008-03-19T06:32:00.000-07:00

$ cleanlinks

this is supported on few flavor of Linux (XFree86), you can use this script.

#/bin/bash
DIR="$1"
[ -d $DIR ] && [ $(ls -l $DIR | wc -l) -eq 1 ] && rmdir $DIR || :

$ script.sh dir1

Removing blank line from file

2008-03-19T06:20:00.000-07:00

Using sed.
$ sed '/^$/d' withblankline.txt > withoutblankline.txt
Using grep.
$ grep -v '^$' withblankline.txt > withoutblankline.txt

Use following for loop (shell script) to remove all blank lines from all files stored in /home/prabhat/data directory:

#!/bin/sh
files="/home/prabhat/data/*.txt"
for f in $files
do
sed '/^$/d' $i > $i.out
mv $i.out $i
done

How to check reverse DNS

2008-03-19T05:49:00.000-07:00

The DNS is used to determine what IP address is associated with a given host name, so to reverse resolve a known IP address is to lookup what the associated host name for it. A reverse lookup is often referred to simply as reverse resolving, or more specifically reverse DNS lookups.

Use of reverse DNS.

Anti-spam
Network troubleshooting
Avoid spammers and phishers using a forward confirmed reverse DNS etc

Check reverse DNS for given IP address under Linux or Windows XP/Server 2003.
You can use standard UNIX / Linux utilities such as nslookup, dig or hosts to find out reverse DNS of a given IP address.

eg; under Linux/UNIX/
$ host 66.65.123.34
or
dig -x IP (66.65.123.34)
and under Linux/UNIX/Windows
nslookup 66.65.123.34

ssh warning

2008-01-07T01:36:00.000-08:00

This is not a big thing, but sometime this small things matter a lot.

You can put a message on remote ssh login for user or it will just act as a warning for a defaulters.

Steps:
1. Open a file using : emacs /etc/motd and write a messge you want
e.g.
********************************************************************
* *
* This system is for the use of authorized users only. Usage of *
* this system may be monitored and recorded by system personnel. *
* *
* Anyone using this system expressly consents to such monitoring *
* and is advised that if such monitoring reveals possible *
* evidence of criminal activity, system personnel may provide the *
* evidence from such monitoring to law enforcement officials. *
* *
********************************************************************
==========================================================
Unauthorized access to this machine is prohibited
Press if you are not an authorized user

2. Save the file and exist.
3. Restart the ssh service using. #service sshd restart

Install GNU/Linux without any CD, floppy, USB-key, nor any other removable media

2007-10-24T23:56:00.000-07:00

Install GNU/Linux without any CD, floppy, USB-key, nor any other removable media

I have a server running CentOS 4 with internet on it. This server is without any CD Drive, floppy Drive, USB-key, nor any other removable media and its Remote server.

How do i install CentOS 5 on it? I have searched in various forums, posted in various forums.
After a long hour I have came to solution.

I am writting here what excataly I have done.

Since, I have net connection on current system,

1. I have downloaded ISO image of centos 5 (1 CD only)
2. mkdir /mnt/tmp
3. mount -o loop CentOS-5.0-x86_64-bin-1of7.iso /mnt/tmp
4. cp -a /mnt/tmp/isolinux/vmlinuz /boot/vmlinuzMy
5. cp -a /mnt/tmp/isolinux/initrd.img /boot/initrdMy.img
6. umount /mnt/tmp
7. Add entry to your /etc/grub.conf
title Fedora Core 2 Installation
root (hd0,0) kernel /boot/vmlinuzMy
initrd /boot/initrdMy.img
8. Reboot system
9. Now your is booted and ready for new installation.
10. Select installation method you want (FTP, HTTP or NFS probably). I have selected http
11. Give your Name server and ipdetails.
12. Give site & Dir (I have given)
mirror.stanford.edu
/yum/pub/centos/5/os/x86_64/

I think its ok to install now :)

System Security Monitor root access

2007-10-18T02:38:00.000-07:00

Security of the server always come first , especially web server which access by the worldwide and hackers always keep eyes on those server's.
So, its good if you have system which automatically send a email notification via email whenever someone logs in as root on your system.
To configure the automatic email alert notification to a email address on each incident of root log on on the server, use the following guide.
Login to the server via SSH using as root ID.

Ensure that you’re at home directory of root. The open up the .bash_profile for editing using pico or vi by typing one of the following commands at linux command shell:
pico .bash_profilevi .bash_profile
Scroll down to the end of the file and add the following line:
echo 'ALERT - Root Shell Access on:' `date` `who` mail -s "Alert: Root Access from `who cut -d"(" -f2 cut -d")" -f1`" user@example.com
Replace user@email.com with the actual email account address that you want to the root access alert notification been sent to. Note that you can change the text contains in the email alert too. The text starting with first ALERT is written as email body, and you can add in other info such as host name or change the wordings. The second Alert is the email title which you can change to your own too.

Note that you can change the text contains in the email alert too. The text starting with first ALERT is written as email body, and you can add in other info such as host name or change the wordings. The second Alert is the email title which you can change to your own too.

Now logout and login again as root, you should receive an email alert at your inbox. The security trick should works on most popular flavor of Linux such as RedHat, CentOS, Ubuntu, FreeBSD and etc.

Trademark Problem

2007-09-15T01:53:00.000-07:00

I am Technical guy I am totally unaware of trademark related things.

But you must be thinking in admin Linux blog I am talking about trademark what happen to this guy. Believe me I am not crazy.

One day I got this comment:

"
Dear Mr. Kumar,Your "Open Hearts" header presents a bit of a problem for us as we hold the international trademark for "Open hearts. Open minds. Open doors." We ask you, kindly, to please remove this statement from your blog. If you have any questions, please contact us at im@umcom.org. --The United Methodist Church.
"

"Open Hearts, Open Minds, Open Doors... Open Source" yes this Title Description is trademark to someone.

Now just I reshuffled this and made it to “Open Minds, Open Hearts, Open Doors... Open Source"

Hope this will not create any problem.

Backup strategy

2007-09-13T04:26:00.000-07:00

Backup strategy:
In computer world most important thing is DATA. No system can 100% secure or fault tolerance, so we always worry & care about data.
Various organization follows different back strategy. Like daily, weekly, monthly or yearly and at different levels.
Once my CEO asked me to design the backup system, and then I study various backup system, algorithms and architectures.

Then I come up with this backup architecture

Level 0 - Hard disk backup using RAID -1 mirroring, its purpose is to use in the case of hardware failure.

Level 1 – OS backup using rsnapshot. It is a file system backup utility based on rsyn, using rsnapshot; it is possible to take snapshots of your file systems at different points in time. Its purpose is to use in the case of OS failure.

Level 2 – Application data backup on same data center using Rsync. Its purpose is to use in the case of application failure.

Level 3 – Application data backup at remote location. Its purpose is to use in the case of data center failure.

As you move from level – 0 to level -3, you have to take more revisions like hours 1, day 1, week 1, and year.

I think this is quit good backup strategy.

Xen : virtualization software

2007-06-25T02:57:00.000-07:00

In my previous blog post I was talked about Virtualization.

Now I am writting about XEN , It is software that runs on a host operating system and allows one to run several guest operating systems on top of the host on the same computer hardware at the same time. Xen Software which enabling every server to support multiple virtual servers. It is the fastest and most secure virtualization software.

In Xen virtualization, a thin software layer known as the Xen hypervisor ( open source technology, developed collaboratively by the world’s best engineers at over 20 of the most innovative data center solution vendors, including Intel, AMD, Cisco, Dell, Egenera, HP, IBM, Mellanox, Network Appliance, Novell, Red Hat, SGI, Sun, Unisys, Veritas, Voltaire, and of course, XenSource ) is inserted between the server’s hardware and the operating system.

This provides an abstraction layer that allows each physical server to run one or more “virtual servers,” effectively decoupling the operating system and its applications from the underlying physical server.

In CentOS-5 Its In build

http://www.centos.org/
Some useful Link here
http://www.xensource.com/download/
http://www.xensource.com/media/xen/player.html
http://bits.xensource.com/Xen/docs/user.pdf
http://www.linuxjournal.com/article/8540
http://wiki.xensource.com/xenwiki/XenIntro

Virtualization : Basic Concepts

2007-04-27T22:41:00.000-07:00

Virtualization is the creation of a virtual (rather than actual) version of something, such as an operating system, a server, a storage device or network resources. e.g. hard drive into different partitions, a partition is the logical division of a hard disk drive to create, in effect, two separate hard drives.
Similarly, Operating system virtualization is the use of software to allow a piece of hardware to run multiple operating system images at the same time.

There are three areas of IT where virtualization is making head roads
a) Network virtualization
b) Storage virtualization
c) Server virtualization

Today, virtualization is in the forefront - helping businesses with scalability, security and management of their global IT infrastructure.

Virtualization can host multiple guest operating systems. Each guest operating system runs in its own domain, Virtualization schedules virtual CPUs within the virtual machines to make the best use of the available physical CPUs. Each guest operating systems handles its own applications. These guest operating systems schedule each application accordingly.

You can deploy Virtualization in one of two choices: full virtualization or Para virtualization.

Full virtualization provides total abstraction of the underlying physical system and creates a new virtual system in which the guest operating systems can run. No modifications are needed in the guest OS or application (the guest OS or application is not aware of the virtualized environment and runs normally).

Para virtualization requires user modification of the guest operating systems that run on the virtual machines (these guest operating systems are aware that they are running on a virtual machine) and provide near-native performance.

To know more on Virtualization:
Intro to Virtualization,
SearchServerVirtualization.com,
Red Hat Enterprise Linux Virtualization.

Change the default port for SSh user

2007-04-23T22:34:00.000-07:00

Default port for SSH user is 22. But we can change the default port to some another port.
eg. I want to run ssh service on port 99 instead of 22.

Make the following change.

open files
#emacs /etc/ssh/ssh_config
#emacs /etc/ssh/sshd_config

search for port ie is default port = 22

make port = 99

or

You can run on any port, But make sure other service not running on that port.

web Based administration solution for Linux : webmin

2007-04-19T02:17:00.000-07:00

Webmin is an open source web Based administration solution for Linux. The purpose of webmin is to gather the administration of many different tasks and services under a common web interface. The software is perl based and module based which make it adaptable. Most common tasks have a module to administer.

how To Install Webmin:
# cd /usr/src
# wget http://prdownloads.sourceforge.net/webadmin/webmin-1.340-1.noarch.rpm
# rpm -ivh webmin-1.340-1.noarch.rpm

steps to install virtualmin & configure it:
login to webmin: ipaddress:100000
1) webmin -> webmin configuration -> webmin modulesInstall module -> from ftp or http url ->virtual min url is: http://download.webmin.com/download/virtualmin/virtual-server-3.201.gpl.wbm.gz

2) Enable https:yum install mod_sslin virtualmin config enable ssl for websites.
Configuring virtualmin:Enabled the subverion and mysql moduledisabled webalizerPostgreSQL setup and configuration enabled? yes.

3) If this server is going to use PGsqlPostfix configuration from webmin:servers->postfix->virtual domain->domain mapping lookup table: hash:/etc/postfix/virtual General options: Network interface for receiving mail: all [make sure that the hostname is something like sys.abc.comg is you are going to run the acb system here.]

4) dovecot configuration from webmin:webmin -> servers -> dovecot -> user login options -> Password authentication source -> unix shadow filestart at boot: yes

5) Webmin -> webmin configuration -> Ports and addressesaccess the webmin system: and then configure the port (eg 100000) for webmin listen and udp packets.

6) Webmin -> webmin configuration -> ssl encryptionAlso make webmin run on ssh.Enable SSL if available? yesRedirect non-SSL requests to SSL mode? yes

Check your memory !

2007-04-02T05:31:00.000-07:00

We all are always worry about memory weather it is main memory, secondary, or other swap
These are few command which help u to debug your memory.

Simple stat.
#free
#du -h
#df -h

Some complex stat.

#cat /proc/meminfo
#dmesg | grep -n kernel

This command give you files have more than 100M usages.

#find / -size +100M -fprintf /root/big.txt '%-10s %p\n'

find out all failed login attempts via ssh/telnet

2007-02-13T23:22:00.000-08:00

how do I find out all failed login attempts via ssh/telnet?

# grep "authentication failure" /var/log/messages|awk '{ print $13 }' | cut -b7- | sort | uniq -c

find command with examples

2007-02-08T05:07:00.000-08:00

find command all we know but here u find some example,

find . -name "rc.conf" -print
This command will search in the current directory and all sub directories for a file named rc.conf.

find /usr/src -not $ -name "*,v" -o -name ".*,v" $ '{}' \; -print
This command will search in the /usr/src directory and all sub directories. All files that are of the form '*,v' and '.*,v' are excluded. Important arguments to note are:

* -not means the negation of the expression that follows
* $ means the start of a complex expression.
* $ means the end of a complex expression.
* -o means a logical or of a complex expression.
In this case the complex expression is all files like '*,v' or '.*,v'

The above example is shows how to select all file that are not part of the RCS system. This is important when you want go through a source tree and modify all the source files... but ... you don't want to affect the RCS version control files.

find . -exec grep "www.athabasca" '{}' \; -print
This command will search in the current directory and all sub directories.

find . -exec grep -q "www.athabasca" '{}' \; -print

This command is very important for process a series of files that contain a specific string. You can then process each file appropriately. An example is find all html files with the string "www.athabascau.ca". You can then process the files with a sed script to change those occurrances of "www.athabascau.ca" with "intra.athabascau.ca"

Recover Root Password of MYSQL

2007-01-09T03:38:00.000-08:00

Step # 1 : Stop mysql service
# /etc/init.d/mysql stop

Step # 2: Start to MySQL server w/o password:
# mysqld_safe --skip-grant-tables &

Step # 3: Connect to mysql server using mysql client:
# mysql -u root

Step # 4: Setup new MySQL root user password
mysql> use mysql;
mysql> update user set password=PASSWORD("NEW-ROOT-PASSWORD") where User='root';
mysql> flush privileges;
mysql> quit

Step # 5: Stop MySQL Server:
# /etc/init.d/mysql stop

Step # 6: Start MySQL server and test it
# /etc/init.d/mysql start
# mysql -u root -p

Enjoy ;-)

Recover Root Password In Liunx Machine

2007-01-08T22:10:00.000-08:00

Recover Root Password

Probably the simplest way to solve a forgotten root password problem is to boot your system in the single-user mode.
If you are using LILO, at the LILO boot prompt (graphical LILO users can press Ctrl-x to exit the graphical screen and go to the boot: prompt), and then enter:
Code:
linux single
This will make you the "root" user without asking for a password. Once the system has booted, you can change the root password using the password command:
Code:
passwd
GRUB users will follow basically the same steps, except that the GRUB boot loader doesn't have a default boot prompt, but you can choose "e'' when the menu displays to edit the boot parameters. Just select the kernel line for the kernel you want to boot. Go to the end of that line and type "single" as a separate word, then press ENTER to exit the edit mode. Once back at the GRUB screen, press "b" to boot into single user mode.

Backup of DataBase from MYSQL

2007-01-05T00:58:00.000-08:00

Every database admin job is to take DB backup time to time.
there is two method's i am using..
Method 1st :
mysqldump client is a backup program used to dump a database or a collection of databases for backup or transfer to another SQL server. The dump typically contains SQL statements to create the table, populate it, or both. However, mysqldump can also be used to generate files in CSV, other delimited text, or XML format.

The most common use of mysqldump is probably for making a backup of an entire database:
shell> mysqldump db_name > backup-file.sql
You can read the dump file back into the server like this:
shell> mysql db_name < backup-file.sql

Or like this:
shell> mysql -e "source /path-to-backup/backup-file.sql" db_name

mysqldump is also very useful for populating databases by copying data from one MySQL server to another:
shell> mysqldump --opt db_name | mysql --host=remote_host -C db_name

It is possible to dump several databases with one command:
shell> mysqldump --databases db_name1 [db_name2 ...] > my_databases.sql

To dump all databases, use the --all-databases option:
shell> mysqldump --all-databases > all_databases.sql
more u can read

Method 2nd:
find out where ur mysql sotre's the database generally default path will be:
:var/lib/mysql this is datadir=var/lib/mysql which is mention in mysql configuration file.

All the DB's u willfind on mysql dir, so u can copy the mysql dir.
this kind of backup used when u want to uninstall and then again install with different version.
simply move to other location mysql dir, and then copy that.

rm and the 'Argument list too long'

2006-12-26T19:33:00.000-08:00

you wanted to delete all the files in a directory that begins with the word 'temp' bala bala

[prabhat@my-site]# rm temp*
bash: /bin/rm: Argument list too long

ERROR!
This happens when you are trying to delete too many files in a directory at the same time - it seems rm has limits ....

To solve the problem:

Use 'find' to pipe all the matching files to 'rm', one at a time.
[prabhat@my-site]# find . -name 'temp*' | xargs rm

or

[prabhat@my-site]# find . -name 'temp*' - print0 | xargs -0 rm -f

Sort folders by size with one command

2006-12-18T04:03:00.000-08:00

Entire user’s data is under /home, I need to have a list of all the sub folders sorted by the size of the sub folders.
simple command to get the list of sub folders sorted by their size:
du --max-depth=1 /home/ | sort -n -r

Zombie Process

2006-12-01T00:47:00.000-08:00

A process can be sleeping in kernel code. Usually that's because of faulty hardware or a badly written driver- or maybe a little of both. A device that isn't set to the interrupt the driver thinks it is can cause this, for example- the driver is waiting for something its never going to get. The process doesn't ignore your signal- it just never gets it.
A zombie process doesn't react to signals because it's not really a process at all- it's just what's left over after it died. What's supposed to happen is that its parent process was to issue a "wait()" to collect the information about its exit. If the parent doesn't (programming error or just bad programming), you get a zombie. The zombie will go away if its parent dies- it will be "adopted" by init which will do the wait()- so if you see one hanging about, check its parent; if it is init, it will be gone soon, if not the only recourse is to kill the parent..which you may or may not want to do.
* Finally, a process that is being traced (by a debugger, for example) won't react to the KILL either.

We can find out zombie process by :-
Use top or ps command:
# top
OR
# ps aux | awk '{ print $8 " " $2 }' | grep -w Z

How do I kill zombie process?
You cannot kill zombies, as they are already dead. But if you have too many zombies then kill parent process or restart service.

You can kill zombie process using PID obtained from any one of the above command. For example kill zombie proces having PID 4104:
# kill -9 4104
*Please note that kill -9 does not guarantee to kill a zombie process

How do I automate zombie process killing?
Write a script and schedule as a cron job.
for each in `ps jauxww | grep Z | grep -v PID | awk ‘{print $3}’`; do for every in `ps auxw | grep $each | grep cron | awk ‘{print $2}’`; do kill -9 $every; done; done

grep command : Finds text within a file

2006-11-23T22:19:00.000-08:00

The grep command searches for the pattern specified by the Pattern parameter and writes each matching line to standard output.
Examples

0. $ grep '12.00' /home/prabhat/backup/log.txt
This command basically shows how you can use grep to extract lines containing a particular string from a text file.
1. To use a pattern that contains some of the pattern-matching characters *, ^, ?, [, ], $, $, \{, and \}, enter:
grep "^[a-zA-Z]" pgm.s
This displays every line in pgm.s whose first character is a letter.

2. To display all lines that do not match a pattern, enter:
grep -v "^#" pgm.s
This displays every line in pgm.s whose first character is not a # (pound sign).

3. To display all lines in the file1 file that match either the abc or xyz string, enter:
grep -E "abc|xyz" file1

4. To search for a $ (dollar sign) in the file named test2, enter:
grep \\$ test2
The \\ (double backslash) characters are necessary in order to force the shell to pass a \$ (single backslash, dollar sign) to the grep command. The \ (single backslash) character tells the grep command to treat the following character (in this example the $) as a literal character rather than an expression character. Use the fgrep command to avoid the necessity of using escape characters such as the backslash.

^ - match at the beginning of the line
$ - match at the end of the line

Some extra options for grep
-v
Reverses the normal behavior of the grep command - Instead of selecting lines, it rejects the lines that match the given criteria.
-c
It suppresses the normal output and only prints the total count of matching lines instead of the actual lines.
-i
Ignores the case of the text when matching the given pattern.
-w
Checks if the given pattern is a word by itself and not a part of another word. Thus if you search for 'bay' and the word 'baywatch' is present in a file, the particular line containing that word would not be returned in the result.
-l
Only gives the names of the files in which the given pattern was found.
-r
Checks for the given pattern , recursively within the directory that you specify after the -r option

know more
http://www.computerhope.com/unix/ugrep.htm
http://linux.about.com/od/commands/l/blcmdl1_grep.htm
http://www.lowfatlinux.com/linux-grep-manual.html

yum : Yellow dog Updater, Modified

2006-11-22T21:45:00.000-08:00

automated software installation and update for linux
yum is software installation tool for Red hat linux and Fedora Linux. It is a complete software management system. yum is designed to use over network/internet. It does not use CDROM to install packages. If you are using fedora you don't have to install it, it is part of fedora itself.

In the case u don't have yum, then u can download yum from project home page http://linux.duke.edu/projects/yum/download.ptml
u can Install using.(it is avaliable in rmp)
# rpm -ivh yup*
U can Configure yum editing /etc/yum.conf
# emacs /etc/yum.conf
Append or edit code as follows:
[base]
name=Fedora Core $releasever - $basearch - Base
baseurl=http://apt.sw.be/fedora/$releasever/en/$basearch/dag
baseurl=http://mirrors.kernel.org/fedora/core/$releasever/$basearch/os
Install GPG signature key with rpm command:
# rpm --import http://dag.wieers.com/packages/RPM-GPG-KEY.dag.txt
and other keys too (if any using above command)
Update your package list:
# yum check-update
Install a new package called test
# yum install test
To update packages
# yum update
To update a single package called test1
# yum update test1
To remove a package called test2
# yum remove test2
To list all packages
# yum list installed
You can search using grep command
# yum list installed | grep test1
Display information on a package called test
# yum info test
To display list of packages for which updates are available
# yum list updates
To clean all cahced packages:
# yum clean packages
To remove all cached packages and old headers:
# yum clean all
Force a fresh download of package
yum clean headers
yum clean oldheaders
To Know more visit :
http://www.die.net/doc/linux/man/man5/yum.conf.5.html http://www.linuxcommand.org/man_pages/yum8.html
http://www.linuxmanpages.com/man8/yum.8.php

Did u know how many virtual web site's ?

2006-11-16T02:45:00.000-08:00

Following command show how many virtual web site's running on ur server.

httpd -t -D DUMP_VHOSTS
or
apache2 -t -D DUMP_VHOSTS

How to keep a computer from answering to ping?

2006-11-11T02:55:00.000-08:00

it is possible to block pings entirely to your machine..

a simple
"echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all" will do the trick...

to turn it back on, simply
"echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_all"

How to do backup with tar?

2006-10-28T04:30:00.000-07:00

You can mantain a list of files that you with to backup into a file and tar
it when you wish.

tar czvf tarfile.tar.gz -T list_file

where list_file is a simple list of what you want to include into the tar

i.e:

/etc/smb.conf
/root/myfile
/etc/ppp (all files into the /etc/ppp directory)
/opt/gnome/html/gnome-dev-info.html

How to remove a file with a dash as first character?

2006-10-28T04:22:00.000-07:00

If you accidentally created a file with a - in the beginning then you want to
remove it, you have to do :
rm ./-thefile

Secure Deletion of Data from Magnetic and Solid-State Memory

2006-09-26T01:47:00.000-07:00

Secure Deletion of Data from Magnetic and Solid-State Memory

`shred': Remove files more securely

`shred' overwrites devices or files, to help prevent even very
expensive hardware from recovering the data.

Ordinarily when you remove a file (*note rm invocation::), the data
is not actually destroyed. Only the index listing where the file is
stored is destroyed, and the storage is made available for reuse.
There are undelete utilities that will attempt to reconstruct the index
and can bring the file back if the parts were not reused.
On a busy system with a nearly-full drive, space can get reused in a
few seconds. But there is no way to know for sure. If you have
sensitive data, you may want to be sure that recovery is not possible
by actually overwriting the file with non-sensitive data.

The best way to remove something irretrievably is to destroy the
media it's on with acid, melt it down, or the like. For cheap
removable media like floppy disks, this is the preferred method.
However, hard drives are expensive and hard to melt, so the `shred'
utility tries to achieve a similar effect non-destructively.

`Secure Deletion of Data from Magnetic and Solid-State Memory', from the
proceedings of the Sixth USENIX Security Symposium (San Jose,
California, 22-25 July, 1996).
visit: http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html

So. `shred' a linux command : Remove files more securely

shred [OPTION]... FILE[...]

look man page how to use..

Danger of 'Ctrl+C' on the Web

2006-09-25T22:48:00.000-07:00

We do copy various data by ctrl+c for pasting elsewhere. This copied data is stored in clipboard and is accessible from the net by a combination of Javascripts and ASP.

Just try this:
1) Copy any text by 'ctrl+c'
2) Click the Link: www.sourcecodesworld.com/special/clipboard.asp
You will see the text you copied on the Screen which was accessed by this web page. (Check it out !!)

Do not keep sensitive data (like passwords, reditcard numbers, PIN etc.) in the clipboard while surfing the web. It is extremely easy to extract the text stored in the clipboard to steal your sensitive information.

Be cautious ...

To avoid Clipboard Hack Problem, do the following:
1) In Internet Explorer, Go to Tools -> Internet options -> Security
2) Press Custom level.
3) In the security settings, select disable under Allow paste operations via script and click on 'OK. (Now the contents of your clipboard are safe.)
It doesn't work on firefox.
Happy Surfing......

Recover from rm

2006-09-22T21:48:00.000-07:00

In order to undelete a file, you must know the following things:
• On which device your file was stored
• What kind of file system was used (eg. ext2, reiserFS, vfat)
To find it out, type 'mount | column -t'

Or
echo "DEVICE DIRECTORY FS-TYPE" > tmp; mount | cut -d" " -f1,3,5 | sort >> tmp; cat tmp | column -t | sed -e "1s/.*/`tput smso`&`tput rmso`/"

The output should be something like this:

bash$ mount | column -t
/dev/hda5 on / type ext2 (rw)
proc on /proc type proc (rw)
usbdevfs on /proc/bus/usb type usbdevfs (rw)
devpts on /dev/pts type devpts (rw)
/dev/hda1 on /mnt/windows/C type vfat (rw,noexec,nosuid,nodev)
/dev/hda6 on /mnt/windows/E type vfat (rw,noexec,nosuid,nodev)
/dev/hdc5 on /mnt/oldwin type vfat (rw,noexec,nosuid,nodev)

Now, of which (printed) directory was the directory of your deleted file a subdirectory? E.g. if your file was stored on /home/user , you'll have to look for '/', since no closer match can be found. Found it? Cool, right now it's a piece of cake to find the device on which the file was stored and the filesystem type of the device.

If you really need to undelete a file, that's the way to do it:
grep -a -B[size before] -A[size after] 'text' /dev/[your_partition]
Replace [size before], [size after] and [your_partition] with something meaningfull. Don't know what your partition is? Read the Linux undelete
.g.: If you want to undelete a letter (+- 200 lines) starting with "Hi mum" which was stored on /dev/hda1 you can try:
grep -a -B2 -A200 "Hi mum" /dev/hda1
Make sure you do this as root (System administrator)

Proxy server types and uses for HTTP Server

2006-09-22T21:44:00.000-07:00

Proxy server

In an enterprise that uses the Internet, a proxy server is a server that acts as an intermediary between a workstation user and the Internet so that the enterprise can ensure security, administrative control, and caching service. A proxy server is associated with or part of a gateway server that separates the enterprise network from the outside network and a firewall server that protects the enterprise network from outside intrusion.

Proxy servers receive requests intended for other servers and then act to fulfill, forward, redirect, or reject the requests. Exactly which service is carried out for a particular request is based on a number of factors which include: the proxy server's capabilities, what is requested, information contained in the request, where the request came from, the intended destination, and in some cases, who sent the request.

An advantage of a proxy server is that its cache can serve all users. If one or more Internet sites are frequently requested, these are likely to be in the proxy's cache, which will improve user response time. In fact, there are special servers called cache servers. A proxy can also do logging.

HTTP Server (powered by Apache) has proxy server capabilities built in. Activating these services is simply a matter of configuration. This topic explains three common proxy concepts: forward proxy, reverse proxy, and proxy chaining.

Forward proxy :
A forward proxy is the most common form of a proxy server and is generally used to pass requests from an isolated, private network to the Internet through a firewall. Using a forward proxy, requests from an isolated network, or intranet, can be rejected or allowed to pass through a firewall. Requests may also be fulfilled by serving from cache rather than passing through the Internet. This allows a level of network security and lessens network traffic.
A forward proxy server will first check to make sure a request is valid. If a request is not valid, or not allowed (blocked by the proxy), it will reject the request resulting in the client receiving an error or a redirect. If a request is valid, a forward proxy may check if the requested information is cached. If it is, the forward proxy serves the cached information. If it is not, the request is sent through a firewall to an actual content server which serves the information to the forward proxy. The proxy, in turn, relays this information to the client and may also cache it, for future requests.

Reverse proxy
A reverse proxy is another common form of a proxy server and is generally used to pass requests from the Internet, through a firewall to isolated, private networks. It is used to prevent Internet clients from having direct, unmonitored access to sensitive data residing on content servers on an isolated network, or intranet. If caching is enabled, a reverse proxy can also lessen network traffic by serving cached information rather than passing all requests to actual content servers. Reverse proxy servers may also balance workload by spreading requests across a number of content servers. One advantage of using a reverse proxy is that Internet clients do not know their requests are being sent to and handled by a reverse proxy server. This allows a reverse proxy to redirect or reject requests without making Internet clients aware of the actual content server (or servers) on a protected network.
A reverse proxy server will first check to make sure a request is valid. If a request is not valid, or not allowed (blocked by the proxy), it will not continue to process the request resulting in the client receiving an error or a redirect. If a request is valid, a reverse proxy may check if the requested information is cached. If it is, the reverse proxy serves the cached information. If it is not, the reverse proxy will request the information from the content server and serve it to the requesting client. It also caches the information for future requests.

Proxy chaining
A proxy chain uses two or more proxy servers to assist in server and protocol performance and network security. Proxy chaining is not a type of proxy, but a use of reverse and forward proxy servers across multiple networks. In addition to the benefits to security and performance, proxy chaining allows requests from different protocols to be fulfilled in cases where, without chaining, such requests would not be possible or permitted. For example, a request using HTTP is sent to a server that can only handle FTP requests. In order for the request to be processed, it must pass through a server that can handle both protocols. This can be accomplished by making use of proxy chaining which allows the request to be passed from a server that is not able to fulfill such a request (perhaps due to security or networking issues, or its own limited capabilities) to a server that can fulfill such a request.
The first proxy server in a chain will check to make sure a request is valid. If a request is not valid, or not allowed (blocked by the proxy), it will reject the request resulting in the client receiving an error or a redirect. If a request is valid, the proxy may check if the requested information is cached and simply serve it from there. If the requested information is not in cache, the proxy will pass the request on to the next proxy server in the chain. This server also has the ability to fulfill, forward, redirect, or reject the request. If it acts to forward the request then it too passes the request on to yet another proxy server. This process is repeated until the request reaches the last proxy server in the chain. The last server in the chain is required to handle the request by contacting the content server, using whatever protocol is required, to obtain the information. The information is then relayed back through the chain until it reaches the requesting client.

The Squid web caching proxy server

Users configure their web browsers to use the Squid proxy server instead of going to the web directly. The Squid server then checks its web cache for the web information requested by the user. It will return any matching information that finds in its cache, and if not, it will go to the web to find it on behalf of the user. Once it finds the information, it will populate its cache with it and also forward it to the user's web browser.
As you can see, this reduces the amount of data accessed from the web. Another advantage is that you can configure your firewall to only accept HTTP web traffic from the Squid server and no one else. Squid can then be configured to request usernames and passwords for each user that users its services. This provides simple access control to the Internet

Some administration commands

2006-07-25T01:15:00.000-07:00

cat /var/log/secure
(as root) Inspect the important system log. It is really a good idea to do it from time to time if you use Internet access.

watch -n 60 my_command
Execute my_command repeatedly at 60-second intervals (the default interval is 2 seconds).

more u can access.
http://linux.about.com/od/linux101/l/blnewbie5_12.htm

Admin Unix

2006-07-25T01:08:00.000-07:00

This blog for System Administrator of UNIX/LINUX.
This wll help u in Technical and Non- Technical Aspect.