Linux Internals

Wednesday, April 30, 2014

Seagate Kinetic Open Storage

It's primarily for Object Storage in which drive is key/value server with Ethernet connectivity.

The Kinetic Drive API provides following capability
1) Kinetic Key-Value Access
2) Key Ordering and Iterators
3) Cluster Management
4) Drive Management
5) Security Management

Since storage are no longer aggregated behind a server, this enable Software defined storage in which any storage any drive can be provisioned to any process in the cluster.

1st Generation Kinetic Drive Limitation

4KB Key and 1 MB Values.. (Drive Management handles the limitation)
2 1GB ethernet over SGMII (Mechanically identical as SAS connector).

So what is Kinetic ??
1) A new classs of key/value Ethernet Drives + Open API and series of libraries

Drives talks in keys and value as opposed to blocks. They do get, put and deletes.

Drive efficiently manages
1) Managing key ordering
2) QOS
3) Policy based 'drive-2-drive' data migration.
4) Handling of partial device failures and other management.
5) Data Security

This new model where legacy software and hardware are removed and thus storage can truly be disaggregated from compute, so Racks will be more dense, fans are minimized.

Kinetic drive are native key/value stores. This shifts the burden of maintaining the space mapping from filesystem to the drive itself. Application needs to only put and get objects and no longer need to guess the LBA layout or prescribe data location.

Advantage of Kinetic Disk Drive:
        1) Multiple Client with Shared Data.
2) Data Migration between 2 Kinetic Drive.

Kinetic Library API
   a) Admin API : getLog, setSecurity, setup
   b) Client API : Synchronous, Asynchronous, put, get, delete, getNext, getPrevious, getKeyRange, getMetadata.

Kinetic Simulator
    Simulator API provides a simulator boot-strap class used for application to start a new Instance of the simulator as a drive.
    Default Port: 8123 as service port

Simulator Setup
==========
1) Download Eclipse IDE from
http://www.eclipse.org/downloads/packages/eclipse-ide-java-developer/keplersr2

2) extract eclipse-java-kepler-SR2-linux-gtk-x86_64.tar.gz

3) Start eclipse

4) Import Kinetic Git hub repositories... (Make sure you have account with Seagate to access Kinetic Git Hub)

      https://github.com/Seagate/kinetic-java.git
      and
      https://github.com/Seagate/Kinetic-Preview.git

Download Apache-Maven
http://maven.apache.org/download.cgi

Run mvn clean package in Kinetic Directory.

Set classpath environment variable.. make sure set jar file created by maven.

export CLASSPATH=/home/ssd/kinetic/Kinetic-Preview/lib/Kinetic-0.6.0.1-SNAPSHOT.jar:.:/home/ssd/kinetic/Kinetic-Preview/target/kinetic-0.6.0.1-SNAPSHOT-jar-with-dependencies.jar

Saturday, April 19, 2014

mmap caching issue on ARM.

This issue observed on ARM, when doing to map kernel allocated memory

i.e via __get_free_pages.

Kernel module updating some counters are not immediately reflected

to userspace code.

The following code in the userspace program is used to clear the

cache after read operation.

/* community.arm.com */

void clear_arm_cache(const char *start_addr , const char *end_addr)
{ 
 const int syscall = 0xf0002;
 __asm __volatile (
  "mov  r0, %0\n"   
  "mov  r1, %1\n"
  "mov  r7, %2\n"
  "mov     r2, #0x0\n"
  "svc     0x00000000\n"
  :
  : "r" (begin), "r" (end), "r" (syscall)
  : "r0", "r1", "r7"
  );
}

Friday, March 21, 2014

MySQL Commands

Friday, March 14, 2014

Uboot To User Process Kernel Functions....

Courtesy: Mohan & Reddy

================

Before we see how the above components work, the following is the call flow of Linux Kernel boot process for arm architecture. This gives a big picture on whole Linux boot process. We use U-boot bootloader.

ARM Linux Boot Process: Big Picture
U-boot:

_start (cpu/arm920t/start.S)

start_code (cpu/arm920t/start.S)

start_armboot (lib_arm/board.c)

board_init (board/kb9202/kb9202.c)

timer_init (cpu/arm920t/at91/timer.c)

serial_init (drivers/serial/at91rm9200_usart.c)

main_loop (lib_arm/board.c)

Now u-boot is up and running and is in u-boot prompt and ready to accept commands. Assume that kernel image is loaded into RAM and issued bootm command.

do_bootm (common/cmd_bootm.c)

bootm_start (common/cmd_bootm.c)

bootm_load_os (common/cmd_bootm.c)

do_bootm_linux (lib_arm/bootm.c)

stext (linux/arch/arm/kernel/head.S)

Control is given to linux.

Linux Kernel:

—————-

stext (arch/arm/kernel/head.S:78)

__lookup_processor_type (arch/arm/kernel/head-common.S:160)

__lookup_machine_type (arch/arm/kernel/head-common.S:211)

__create_page_tables (arch/arm/kernel/head.S:219)

__arm920_setup (arch/arm/mm/proc-arm920.S:389)

__enable_mmu (arch/arm/kernel/head.S:160)

__turn_mmu_on (arch/arm/kernel/head.S:205)

__switch_data (arch/arm/kernel/head-common.S:20)

start_kernel (init/main.c:529)

tick_init(kernel/time/tick-common.c:413)

setup_arch (arch/arm/kernel/setup.c:666)

setup_machine (arch/arm/kernel/setup.c:369)

lookup_machine_type ( )

setup_command_line (init/main.c:408)

build_all_zonelists (mm/page_alloc.c:3031)

parse_args (kernel/params.c:129)

mm_init (init/main.c:516)

mem_init (arch/arm/mm/init.c:528)

kmem_cache_init (mm/slab.c, mm/slob.c, mm/slub.c)

sched_init (kernel/sched.c)

init_IRQ (arch/arm/kernel/irq.c)

init_timers (kernel/timer.c:1713)

hrtimers_init (kernel/hrtimer.c:1741)

softirq_init (kernel/softirq.c:674)

console_init (drivers/char/tty_io.c:3084)

vfs_caches_init (fs/dcache.c:2352)

mnt_init (fs/namespace.c:2308)

init_rootfs ()

init_mount_tree (fs/namespace.c:2285)

do_kern_mount (fs/namespace.c:1053)

set_fs_pwd(fs/fs_struct.c:29)

set_fs_root(fs/fs_struct.c:12)

bdev_cache_init (fs/block_dev.c:465)

chrdev_init (fs/char_dev.c:566)

signals_init (kernel/signal.c:2737)

rest_init (init/main.c:425)

kernel_thread (431, arch/arm/kernel/process.c:388)

kernel_thread() creates a kernel thread and control is given to kernel_init().

kernel_init (431, init/main.c:856)

do_basic_setup (888, init/main.c:787)

init_workqueues (789, kernel/workqueue.c:1204)

driver_init (793, drivers/base/init.c:20)

do_initcalls (796, init/main.c:769) /* Calls all subsytems init functions */

prepare_namespace (906, init/do_mounts.c:366)

initrd_load (399, init/do_mounts_initrd.c:107)

rd_load_image (117, init/do_mounts_rd.c:158) /* if initrd is given */

identify_ramdisk_image (179, init/do_mounts_rd.c:53)

handle_initrd (119, init/do_mounts_initrd.c:37) /*if rd_load_image is success */

mount_block_root (45, init/do_mounts.c:233)

do_mount_root (247, init/do_mounts.:218)

mount_root (417, init/do_mounts.c:334) /* if initrd not given */

mount_block_root (359, init/do_mounts.c:233)

do_mount_root (247, init/do_mounts.c:218)

init_post (915, init/main.c:816)

run_init_process (847, init/main.c:807)

kernel_execve (810, arch/arm/kernel/sys_arm.c:81)

User Space |

————- |

init() /*userspace /sbin/init */

Monday, January 27, 2014

Reading Clock Counters in Processors

There are clock counters on x86, MIPS & ARM.

These clock count register can be accessed from userspace and kernel space.

x86
===
__asm__ __volatile__("rdtsc" : "=A" (tsc));

ARM 11 & higher (mcr and mrc register)
==============
__asm__ __volatile__("mcr p15, 0, %0, c15, c9, 0" ::"r"(0x1)); // Access validation

__asm__ __volatile__("mrc p15, 0, %0, c15, c12, 1" : "=r"(tsc)); //Read Clock Counter

MIPS (mtc & rdhwr register)
====
__asm__ __volatile__("mtc0 %0, $7" :: "r"(0x40000000)); //Enable hw register

__asm__ __volatile__("rdhwr %0, $2" : "=r"(tsc)); //Read Clock Counter

Wednesday, December 18, 2013

Write to block device from kernel...

int
disk_open(dev_minor_t *devinst, const char *path)
{
        int     partnum;
        dev_t   devt;
        struct hd_struct *part;
        unsigned char   *buf = NULL;
        struct bio *bb;
        int     i, j;
        struct page *page;
        unsigned char *d = NULL;
        DECLARE_COMPLETION_ONSTACK(waithdl);

        if ((devt = blk_lookup_devt(path, 0)) == 0) {
                printk(KERN_INFO "blk_lookup_devt %d\n", devt);
                return (-1);
        }
        printk(KERN_INFO "Success blk_lookup_devt %d\n", devt);
        if ((devinst->disk = get_gendisk(devt, &partnum)) == NULL) {
                printk(KERN_INFO "Failed to get generic device ..");
                return (-1);
        }

        if ((part = disk_get_part(devinst->disk, 0)) == NULL) {
                printk(KERN_INFO "failed disk_get_part");
                return (-1);
        }

        if ((devinst->bdev = bdget_disk(devinst->disk, 0)) == NULL) {
                printk(KERN_INFO "failed bdget_disk");
                return (-1);
        }
        return (0);
}

void
disk_close(struct file *filp)
{

}

int
disk_read(dev_minor_t *devinst, unsigned long long offset,
                                       unsigned char *data, unsigned int size)
{
        struct page *page;
        unsigned char *d = NULL;
        DECLARE_COMPLETION_ONSTACK(waithdl);
        unsigned char   *buf = NULL;
        struct bio *bb;
        int     ret = 0;

        printk(KERN_INFO "Enter %s\n", __FUNCTION__);
        if (data == NULL) {
                page = alloc_page(GFP_KERNEL);
                d = page_address(page);
                memset(d, 0x00, 4096);
        } else {
                d = data;
                memset(d, 0x00, 4096);
        }

        bb = bio_map_kern(devinst->disk->queue, d, 4096, GFP_KERNEL);
        if (IS_ERR(bb)) {
                printk(KERN_ALERT "FAiled to map kernel memory");
                return (-1);
        }
        //bb->bi_sector = offset >> 9;
        bb->bi_sector = 0;
        printk(KERN_INFO "offset %d size %d\n", bb->bi_sector, size);
        bb->bi_bdev = devinst->bdev->bd_contains;
        if (bb->bi_bdev == NULL) {
                bb->bi_bdev = blkdev_get_by_dev(devinst->bdev->bd_dev, FMODE_READ|FMODE_WRITE, NULL);
        }
        if (bb->bi_bdev == NULL) {
                printk(KERN_ALERT "Doesn't have blk device ...\n");
                return (-1);
        }
        bb->bi_end_io = callback_fn;
        bb->bi_private = &waithdl;
        bio_get(bb);

        submit_bio(READ_SYNC, bb); //read operation
        wait_for_completion(&waithdl);

        if (!bio_flagged(bb, BIO_UPTODATE)) {
               printk(KERN_ALERT "Failed ...");
                ret = -1;
        }
        bio_put(bb);
        printk(KERN_INFO "Data read %x%x%x%x", d[0],d[1],d[2],d[3]);
#if 0
        if (data)
                memcpy(data, d, 4096);
#endif
        return (ret);
}

int
disk_write(dev_minor_t *devinst, unsigned long long offset,
                                     unsigned char *data, unsigned int size)
{
        struct page *page;
        unsigned char *d = NULL;
        DECLARE_COMPLETION_ONSTACK(waithdl);
        unsigned char   *buf = NULL;
        struct bio *bb;
        struct bio_vec bio_vec;

        int     ret = 0;

        printk(KERN_INFO "Enter %s\n", __FUNCTION__);

        bb = bio_map_kern(devinst->disk->queue, data, 4096, GFP_KERNEL);
        if (IS_ERR(bb)) {
                printk(KERN_ALERT "FAiled to map kernel memory");
                return (-1);
        }
        bb->bi_sector = 1;
        bb->bi_bdev = devinst->bdev->bd_contains;
        if (bb->bi_bdev == NULL) {
                bb->bi_bdev = blkdev_get_by_dev(devinst->bdev->bd_dev, FMODE_READ|FMODE_WRITE, NULL);
        }
        if (bb->bi_bdev == NULL) {
                printk(KERN_ALERT "Doesn't have blk device ...\n");
                return (-1);
        }
        bb->bi_end_io = callback_fn;
        bb->bi_private = &waithdl;

        bio_get(bb);
        submit_bio(WRITE_SYNC, bb);

        wait_for_completion(&waithdl);

        if (!test_bit(BIO_UPTODATE, &bb->bi_flags)) {
                bio_put(bb);
                ret = -1;
        }
        bio_put(bb);
        return (ret);
}

/* backup code */
page_read() {

        bio_init(&bio);
        bio.bi_io_vec = &bio_vec;
        bio_vec.bv_page = p;
        bio_vec.bv_len = size;
        bio_vec.bv_offset = 0;
        bio.bi_vcnt = 1;
        bio.bi_idx = 0;
        bio.bi_size = size;
        bio.bi_bdev = bdev->bd_contains;
        bio.bi_sector = sect;
        init_completion(&complete);
        bio.bi_private = &complete;
        bio.bi_end_io = rq_complete;

        //submit_bio(WRITE_SYNC, &bio);
        submit_bio(READ_SYNC, &bio);
        wait_for_completion(&complete);

        if (test_bit(BIO_UPTODATE, &bio.bi_flags)) {
               printk(KERN_INFO "Read BIO OK 0x%x%x%x%x\n", d[0],d[1],d[2],d[3]);
        } else {
                printk(KERN_INFO "BIO ERROR\n");
        }
}

Monday, December 9, 2013

How to write to a file from a kernel module

/* The below code, can be used to create a file and write data to a file using vfs call.
*/

#include <linux/fs.h>
#include <asm/segment.h>
#include <asm/uaccess.h>
#include <linux/buffer_head.h>

struct file *
driver_file_open(const char *path, int flags, int mode)
{
        struct file *filp = NULL;
        mm_segment_t    oldfs;
        oldfs   = get_fs();
        set_fs(get_ds());
        filp = filp_open(path, O_CREAT|O_RDWR, S_IRWXU|S_IRWXG|S_IRWXO);
        set_fs(oldfs);
        return (filp);
}

void
driver_file_close(struct file *filp)
{
        filp_close(filp, NULL);
}

int
driver_file_write(struct file *file, unsigned long long offset, unsigned char *data, unsigned int size)
{
        int     ret;
        mm_segment_t    oldfs;
        loff_t pos = offset;
        oldfs   = get_fs();
        set_fs(get_ds());

        //vfs_setpos(file, pos, pos + PAGE_SIZE);
        //Workaround for vfs_setpos, not implemented on my version of linux.
        spin_lock(&file->f_lock);
        file->f_pos = pos;
        //file->f_version = 0;
        printk(KERN_INFO "set position to %llx\n", pos);
        spin_unlock(&file->f_lock);

        ret = vfs_write(file, data, size, &pos);
        //vfs_fsync(file, 0);
        set_fs(oldfs);
        return (ret);
}

int
driver_file_read(struct file *file, unsigned long long offset, unsigned char *data, unsigned int size)
{
        int     ret;
        mm_segment_t    oldfs;
        loff_t pos = offset;
        oldfs   = get_fs();
        set_fs(get_ds());

        //vfs_setpos(file, pos, pos + PAGE_SIZE);
        //Workaround for vfs_setpos, not implemented on my version of linux.
        spin_lock(&file->f_lock);
        file->f_pos = pos;
        //file->f_version = 0;
        printk(KERN_INFO "set position to read %llx\n", pos);
        spin_unlock(&file->f_lock);

        ret = vfs_read(file, data, size, &pos);
        //vfs_fsync(file, 0);
        set_fs(oldfs);
        return (ret);
}