github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-06-08 14:42:37 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

Merge remote-tracking branch 'origin/v3.10/topic/zram' into linux-linaro-lsk

Browse Source 
Conflicts:
	mm/Kconfig
	mm/Makefile
This commit is contained in:
Alex Shi 2015-05-12 14:55:30 +08:00
commit 1f8fdf83fe
37 changed files with 4306 additions and 1193 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

62
Documentation/ABI/testing/sysfs-block-zram
View File

@ -42,15 +42,48 @@ Description:
The invalid_io file is read-only and specifies the number of
non-page-size-aligned I/O requests issued to this device.
What: /sys/block/zram<id>/failed_reads
Date: February 2014
Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Description:
The failed_reads file is read-only and specifies the number of
failed reads happened on this device.
What: /sys/block/zram<id>/failed_writes
Date: February 2014
Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Description:
The failed_writes file is read-only and specifies the number of
failed writes happened on this device.
What: /sys/block/zram<id>/max_comp_streams
Date: February 2014
Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Description:
The max_comp_streams file is read-write and specifies the
number of backend's zcomp_strm compression streams (number of
concurrent compress operations).
What: /sys/block/zram<id>/comp_algorithm
Date: February 2014
Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Description:
The comp_algorithm file is read-write and lets to show
available and selected compression algorithms, change
compression algorithm selection.
What: /sys/block/zram<id>/notify_free
Date: August 2010
Contact: Nitin Gupta <ngupta@vflare.org>
Description:
The notify_free file is read-only and specifies the number of
swap slot free notifications received by this device. These
notifications are send to a swap block device when a swap slot
is freed. This statistic is applicable only when this disk is
being used as a swap disk.
The notify_free file is read-only. Depending on device usage
scenario it may account a) the number of pages freed because
of swap slot free notifications or b) the number of pages freed
because of REQ_DISCARD requests sent by bio. The former ones
are sent to a swap block device when a swap slot is freed, which
implies that this disk is being used as a swap disk. The latter
ones are sent by filesystem mounted with discard option,
whenever some data blocks are getting discarded.
What: /sys/block/zram<id>/discard
Date: August 2010
@ -97,3 +130,22 @@ Description:
efficiency can be calculated using compr_data_size and this
statistic.
Unit: bytes
What: /sys/block/zram<id>/mem_used_max
Date: August 2014
Contact: Minchan Kim <minchan@kernel.org>
Description:
The mem_used_max file is read/write and specifies the amount
of maximum memory zram have consumed to store compressed data.
For resetting the value, you should write "0". Otherwise,
you could see -EINVAL.
Unit: bytes
What: /sys/block/zram<id>/mem_limit
Date: August 2014
Contact: Minchan Kim <minchan@kernel.org>
Description:
The mem_limit file is read/write and specifies the maximum
amount of memory ZRAM can use to store the compressed data. The
limit could be changed in run time and "0" means disable the
limit. No limit is the initial state. Unit: bytes

129
Documentation/blockdev/zram.txt Normal file
View File

@ -0,0 +1,129 @@
zram: Compressed RAM based block devices
----------------------------------------
* Introduction
The zram module creates RAM based block devices named /dev/zram<id>
(<id> = 0, 1, ...). Pages written to these disks are compressed and stored
in memory itself. These disks allow very fast I/O and compression provides
good amounts of memory savings. Some of the usecases include /tmp storage,
use as swap disks, various caches under /var and maybe many more :)
Statistics for individual zram devices are exported through sysfs nodes at
/sys/block/zram<id>/
* Usage
Following shows a typical sequence of steps for using zram.
1) Load Module:
modprobe zram num_devices=4
This creates 4 devices: /dev/zram{0,1,2,3}
(num_devices parameter is optional. Default: 1)
2) Set max number of compression streams
Compression backend may use up to max_comp_streams compression streams,
thus allowing up to max_comp_streams concurrent compression operations.
By default, compression backend uses single compression stream.
Examples:
#show max compression streams number
cat /sys/block/zram0/max_comp_streams
#set max compression streams number to 3
echo 3 > /sys/block/zram0/max_comp_streams
Note:
In order to enable compression backend's multi stream support max_comp_streams
must be initially set to desired concurrency level before ZRAM device
initialisation. Once the device initialised as a single stream compression
backend (max_comp_streams equals to 1), you will see error if you try to change
the value of max_comp_streams because single stream compression backend
implemented as a special case by lock overhead issue and does not support
dynamic max_comp_streams. Only multi stream backend supports dynamic
max_comp_streams adjustment.
3) Select compression algorithm
Using comp_algorithm device attribute one can see available and
currently selected (shown in square brackets) compression algortithms,
change selected compression algorithm (once the device is initialised
there is no way to change compression algorithm).
Examples:
#show supported compression algorithms
cat /sys/block/zram0/comp_algorithm
lzo [lz4]
#select lzo compression algorithm
echo lzo > /sys/block/zram0/comp_algorithm
4) Set Disksize
Set disk size by writing the value to sysfs node 'disksize'.
The value can be either in bytes or you can use mem suffixes.
Examples:
# Initialize /dev/zram0 with 50MB disksize
echo $((50*1024*1024)) > /sys/block/zram0/disksize
# Using mem suffixes
echo 256K > /sys/block/zram0/disksize
echo 512M > /sys/block/zram0/disksize
echo 1G > /sys/block/zram0/disksize
Note:
There is little point creating a zram of greater than twice the size of memory
since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the
size of the disk when not in use so a huge zram is wasteful.
5) Set memory limit: Optional
Set memory limit by writing the value to sysfs node 'mem_limit'.
The value can be either in bytes or you can use mem suffixes.
In addition, you could change the value in runtime.
Examples:
# limit /dev/zram0 with 50MB memory
echo $((50*1024*1024)) > /sys/block/zram0/mem_limit
# Using mem suffixes
echo 256K > /sys/block/zram0/mem_limit
echo 512M > /sys/block/zram0/mem_limit
echo 1G > /sys/block/zram0/mem_limit
# To disable memory limit
echo 0 > /sys/block/zram0/mem_limit
6) Activate:
mkswap /dev/zram0
swapon /dev/zram0
mkfs.ext4 /dev/zram1
mount /dev/zram1 /tmp
7) Stats:
Per-device statistics are exported as various nodes under
/sys/block/zram<id>/
disksize
num_reads
num_writes
invalid_io
notify_free
discard
zero_pages
orig_data_size
compr_data_size
mem_used_total
mem_used_max
8) Deactivate:
swapoff /dev/zram0
umount /dev/zram1
9) Reset:
Write any positive value to 'reset' sysfs node
echo 1 > /sys/block/zram0/reset
echo 1 > /sys/block/zram1/reset
This frees all the memory allocated for the given device and
resets the disksize to zero. You must set the disksize again
before reusing the device.
Nitin Gupta
ngupta@vflare.org

2
drivers/block/Kconfig
View File

@ -105,6 +105,8 @@ source "drivers/block/paride/Kconfig"
source "drivers/block/mtip32xx/Kconfig"
source "drivers/block/zram/Kconfig"
config BLK_CPQ_DA
tristate "Compaq SMART2 support"
depends on PCI && VIRT_TO_BUS

1
drivers/block/Makefile
View File

@ -41,6 +41,7 @@ obj-$(CONFIG_BLK_DEV_RBD) += rbd.o
obj-$(CONFIG_BLK_DEV_PCIESSD_MTIP32XX) += mtip32xx/
obj-$(CONFIG_BLK_DEV_RSXX) += rsxx/
obj-$(CONFIG_ZRAM) += zram/
nvme-y := nvme-core.o nvme-scsi.o
swim_mod-y := swim.o swim_asm.o

11
drivers/staging/zram/Kconfig → drivers/block/zram/Kconfig
View File

@ -14,7 +14,16 @@ config ZRAM
disks and maybe many more.
See zram.txt for more information.
Project home: <https://compcache.googlecode.com/>
config ZRAM_LZ4_COMPRESS
bool "Enable LZ4 algorithm support"
depends on ZRAM
select LZ4_COMPRESS
select LZ4_DECOMPRESS
default n
help
This option enables LZ4 compression algorithm support. Compression
algorithm can be changed using `comp_algorithm' device attribute.
config ZRAM_DEBUG
bool "Compressed RAM block device debug support"

5
drivers/block/zram/Makefile Normal file
View File

@ -0,0 +1,5 @@
zram-y := zcomp_lzo.o zcomp.o zram_drv.o
zram-$(CONFIG_ZRAM_LZ4_COMPRESS) += zcomp_lz4.o
obj-$(CONFIG_ZRAM) += zram.o

353
drivers/block/zram/zcomp.c Normal file
View File

@ -0,0 +1,353 @@
/*
* Copyright (C) 2014 Sergey Senozhatsky.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
#include <linux/kernel.h>
#include <linux/string.h>
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/wait.h>
#include <linux/sched.h>
#include "zcomp.h"
#include "zcomp_lzo.h"
#ifdef CONFIG_ZRAM_LZ4_COMPRESS
#include "zcomp_lz4.h"
#endif
/*
* single zcomp_strm backend
*/
struct zcomp_strm_single {
struct mutex strm_lock;
struct zcomp_strm *zstrm;
};
/*
* multi zcomp_strm backend
*/
struct zcomp_strm_multi {
/* protect strm list */
spinlock_t strm_lock;
/* max possible number of zstrm streams */
int max_strm;
/* number of available zstrm streams */
int avail_strm;
/* list of available strms */
struct list_head idle_strm;
wait_queue_head_t strm_wait;
};
static struct zcomp_backend *backends[] = {
&zcomp_lzo,
#ifdef CONFIG_ZRAM_LZ4_COMPRESS
&zcomp_lz4,
#endif
NULL
};
static struct zcomp_backend *find_backend(const char *compress)
{
int i = 0;
while (backends[i]) {
if (sysfs_streq(compress, backends[i]->name))
break;
i++;
}
return backends[i];
}
static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
{
if (zstrm->private)
comp->backend->destroy(zstrm->private);
free_pages((unsigned long)zstrm->buffer, 1);
kfree(zstrm);
}
/*
* allocate new zcomp_strm structure with ->private initialized by
* backend, return NULL on error
*/
static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp)
{
struct zcomp_strm *zstrm = kmalloc(sizeof(*zstrm), GFP_KERNEL);
if (!zstrm)
return NULL;
zstrm->private = comp->backend->create();
/*
* allocate 2 pages. 1 for compressed data, plus 1 extra for the
* case when compressed size is larger than the original one
*/
zstrm->buffer = (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, 1);
if (!zstrm->private || !zstrm->buffer) {
zcomp_strm_free(comp, zstrm);
zstrm = NULL;
}
return zstrm;
}
/*
* get idle zcomp_strm or wait until other process release
* (zcomp_strm_release()) one for us
*/
static struct zcomp_strm *zcomp_strm_multi_find(struct zcomp *comp)
{
struct zcomp_strm_multi *zs = comp->stream;
struct zcomp_strm *zstrm;
while (1) {
spin_lock(&zs->strm_lock);
if (!list_empty(&zs->idle_strm)) {
zstrm = list_entry(zs->idle_strm.next,
struct zcomp_strm, list);
list_del(&zstrm->list);
spin_unlock(&zs->strm_lock);
return zstrm;
}
/* zstrm streams limit reached, wait for idle stream */
if (zs->avail_strm >= zs->max_strm) {
spin_unlock(&zs->strm_lock);
wait_event(zs->strm_wait, !list_empty(&zs->idle_strm));
continue;
}
/* allocate new zstrm stream */
zs->avail_strm++;
spin_unlock(&zs->strm_lock);
zstrm = zcomp_strm_alloc(comp);
if (!zstrm) {
spin_lock(&zs->strm_lock);
zs->avail_strm--;
spin_unlock(&zs->strm_lock);
wait_event(zs->strm_wait, !list_empty(&zs->idle_strm));
continue;
}
break;
}
return zstrm;
}
/* add stream back to idle list and wake up waiter or free the stream */
static void zcomp_strm_multi_release(struct zcomp *comp, struct zcomp_strm *zstrm)
{
struct zcomp_strm_multi *zs = comp->stream;
spin_lock(&zs->strm_lock);
if (zs->avail_strm <= zs->max_strm) {
list_add(&zstrm->list, &zs->idle_strm);
spin_unlock(&zs->strm_lock);
wake_up(&zs->strm_wait);
return;
}
zs->avail_strm--;
spin_unlock(&zs->strm_lock);
zcomp_strm_free(comp, zstrm);
}
/* change max_strm limit */
static bool zcomp_strm_multi_set_max_streams(struct zcomp *comp, int num_strm)
{
struct zcomp_strm_multi *zs = comp->stream;
struct zcomp_strm *zstrm;
spin_lock(&zs->strm_lock);
zs->max_strm = num_strm;
/*
* if user has lowered the limit and there are idle streams,
* immediately free as much streams (and memory) as we can.
*/
while (zs->avail_strm > num_strm && !list_empty(&zs->idle_strm)) {
zstrm = list_entry(zs->idle_strm.next,
struct zcomp_strm, list);
list_del(&zstrm->list);
zcomp_strm_free(comp, zstrm);
zs->avail_strm--;
}
spin_unlock(&zs->strm_lock);
return true;
}
static void zcomp_strm_multi_destroy(struct zcomp *comp)
{
struct zcomp_strm_multi *zs = comp->stream;
struct zcomp_strm *zstrm;
while (!list_empty(&zs->idle_strm)) {
zstrm = list_entry(zs->idle_strm.next,
struct zcomp_strm, list);
list_del(&zstrm->list);
zcomp_strm_free(comp, zstrm);
}
kfree(zs);
}
static int zcomp_strm_multi_create(struct zcomp *comp, int max_strm)
{
struct zcomp_strm *zstrm;
struct zcomp_strm_multi *zs;
comp->destroy = zcomp_strm_multi_destroy;
comp->strm_find = zcomp_strm_multi_find;
comp->strm_release = zcomp_strm_multi_release;
comp->set_max_streams = zcomp_strm_multi_set_max_streams;
zs = kmalloc(sizeof(struct zcomp_strm_multi), GFP_KERNEL);
if (!zs)
return -ENOMEM;
comp->stream = zs;
spin_lock_init(&zs->strm_lock);
INIT_LIST_HEAD(&zs->idle_strm);
init_waitqueue_head(&zs->strm_wait);
zs->max_strm = max_strm;
zs->avail_strm = 1;
zstrm = zcomp_strm_alloc(comp);
if (!zstrm) {
kfree(zs);
return -ENOMEM;
}
list_add(&zstrm->list, &zs->idle_strm);
return 0;
}
static struct zcomp_strm *zcomp_strm_single_find(struct zcomp *comp)
{
struct zcomp_strm_single *zs = comp->stream;
mutex_lock(&zs->strm_lock);
return zs->zstrm;
}
static void zcomp_strm_single_release(struct zcomp *comp,
struct zcomp_strm *zstrm)
{
struct zcomp_strm_single *zs = comp->stream;
mutex_unlock(&zs->strm_lock);
}
static bool zcomp_strm_single_set_max_streams(struct zcomp *comp, int num_strm)
{
/* zcomp_strm_single support only max_comp_streams == 1 */
return false;
}
static void zcomp_strm_single_destroy(struct zcomp *comp)
{
struct zcomp_strm_single *zs = comp->stream;
zcomp_strm_free(comp, zs->zstrm);
kfree(zs);
}
static int zcomp_strm_single_create(struct zcomp *comp)
{
struct zcomp_strm_single *zs;
comp->destroy = zcomp_strm_single_destroy;
comp->strm_find = zcomp_strm_single_find;
comp->strm_release = zcomp_strm_single_release;
comp->set_max_streams = zcomp_strm_single_set_max_streams;
zs = kmalloc(sizeof(struct zcomp_strm_single), GFP_KERNEL);
if (!zs)
return -ENOMEM;
comp->stream = zs;
mutex_init(&zs->strm_lock);
zs->zstrm = zcomp_strm_alloc(comp);
if (!zs->zstrm) {
kfree(zs);
return -ENOMEM;
}
return 0;
}
/* show available compressors */
ssize_t zcomp_available_show(const char *comp, char *buf)
{
ssize_t sz = 0;
int i = 0;
while (backends[i]) {
if (sysfs_streq(comp, backends[i]->name))
sz += scnprintf(buf + sz, PAGE_SIZE - sz - 2,
"[%s] ", backends[i]->name);
else
sz += scnprintf(buf + sz, PAGE_SIZE - sz - 2,
"%s ", backends[i]->name);
i++;
}
sz += scnprintf(buf + sz, PAGE_SIZE - sz, "\n");
return sz;
}
bool zcomp_set_max_streams(struct zcomp *comp, int num_strm)
{
return comp->set_max_streams(comp, num_strm);
}
struct zcomp_strm *zcomp_strm_find(struct zcomp *comp)
{
return comp->strm_find(comp);
}
void zcomp_strm_release(struct zcomp *comp, struct zcomp_strm *zstrm)
{
comp->strm_release(comp, zstrm);
}
int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
const unsigned char *src, size_t *dst_len)
{
return comp->backend->compress(src, zstrm->buffer, dst_len,
zstrm->private);
}
int zcomp_decompress(struct zcomp *comp, const unsigned char *src,
size_t src_len, unsigned char *dst)
{
return comp->backend->decompress(src, src_len, dst);
}
void zcomp_destroy(struct zcomp *comp)
{
comp->destroy(comp);
kfree(comp);
}
/*
* search available compressors for requested algorithm.
* allocate new zcomp and initialize it. return compressing
* backend pointer or ERR_PTR if things went bad. ERR_PTR(-EINVAL)
* if requested algorithm is not supported, ERR_PTR(-ENOMEM) in
* case of allocation error.
*/
struct zcomp *zcomp_create(const char *compress, int max_strm)
{
struct zcomp *comp;
struct zcomp_backend *backend;
backend = find_backend(compress);
if (!backend)
return ERR_PTR(-EINVAL);
comp = kzalloc(sizeof(struct zcomp), GFP_KERNEL);
if (!comp)
return ERR_PTR(-ENOMEM);
comp->backend = backend;
if (max_strm > 1)
zcomp_strm_multi_create(comp, max_strm);
else
zcomp_strm_single_create(comp);
if (!comp->stream) {
kfree(comp);
return ERR_PTR(-ENOMEM);
}
return comp;
}

68
drivers/block/zram/zcomp.h Normal file
View File

@ -0,0 +1,68 @@
/*
* Copyright (C) 2014 Sergey Senozhatsky.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
#ifndef _ZCOMP_H_
#define _ZCOMP_H_
#include <linux/mutex.h>
struct zcomp_strm {
/* compression/decompression buffer */
void *buffer;
/*
* The private data of the compression stream, only compression
* stream backend can touch this (e.g. compression algorithm
* working memory)
*/
void *private;
/* used in multi stream backend, protected by backend strm_lock */
struct list_head list;
};
/* static compression backend */
struct zcomp_backend {
int (*compress)(const unsigned char *src, unsigned char *dst,
size_t *dst_len, void *private);
int (*decompress)(const unsigned char *src, size_t src_len,
unsigned char *dst);
void *(*create)(void);
void (*destroy)(void *private);
const char *name;
};
/* dynamic per-device compression frontend */
struct zcomp {
void *stream;
struct zcomp_backend *backend;
struct zcomp_strm *(*strm_find)(struct zcomp *comp);
void (*strm_release)(struct zcomp *comp, struct zcomp_strm *zstrm);
bool (*set_max_streams)(struct zcomp *comp, int num_strm);
void (*destroy)(struct zcomp *comp);
};
ssize_t zcomp_available_show(const char *comp, char *buf);
struct zcomp *zcomp_create(const char *comp, int max_strm);
void zcomp_destroy(struct zcomp *comp);
struct zcomp_strm *zcomp_strm_find(struct zcomp *comp);
void zcomp_strm_release(struct zcomp *comp, struct zcomp_strm *zstrm);
int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
const unsigned char *src, size_t *dst_len);
int zcomp_decompress(struct zcomp *comp, const unsigned char *src,
size_t src_len, unsigned char *dst);
bool zcomp_set_max_streams(struct zcomp *comp, int num_strm);
#endif /* _ZCOMP_H_ */

47
drivers/block/zram/zcomp_lz4.c Normal file
View File

@ -0,0 +1,47 @@
/*
* Copyright (C) 2014 Sergey Senozhatsky.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/lz4.h>
#include "zcomp_lz4.h"
static void *zcomp_lz4_create(void)
{
return kzalloc(LZ4_MEM_COMPRESS, GFP_KERNEL);
}
static void zcomp_lz4_destroy(void *private)
{
kfree(private);
}
static int zcomp_lz4_compress(const unsigned char *src, unsigned char *dst,
size_t *dst_len, void *private)
{
/* return : Success if return 0 */
return lz4_compress(src, PAGE_SIZE, dst, dst_len, private);
}
static int zcomp_lz4_decompress(const unsigned char *src, size_t src_len,
unsigned char *dst)
{
size_t dst_len = PAGE_SIZE;
/* return : Success if return 0 */
return lz4_decompress_unknownoutputsize(src, src_len, dst, &dst_len);
}
struct zcomp_backend zcomp_lz4 = {
.compress = zcomp_lz4_compress,
.decompress = zcomp_lz4_decompress,
.create = zcomp_lz4_create,
.destroy = zcomp_lz4_destroy,
.name = "lz4",
};

17
drivers/block/zram/zcomp_lz4.h Normal file
View File

@ -0,0 +1,17 @@
/*
* Copyright (C) 2014 Sergey Senozhatsky.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
#ifndef _ZCOMP_LZ4_H_
#define _ZCOMP_LZ4_H_
#include "zcomp.h"
extern struct zcomp_backend zcomp_lz4;
#endif /* _ZCOMP_LZ4_H_ */

47
drivers/block/zram/zcomp_lzo.c Normal file
View File

@ -0,0 +1,47 @@
/*
* Copyright (C) 2014 Sergey Senozhatsky.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/lzo.h>
#include "zcomp_lzo.h"
static void *lzo_create(void)
{
return kzalloc(LZO1X_MEM_COMPRESS, GFP_KERNEL);
}
static void lzo_destroy(void *private)
{
kfree(private);
}
static int lzo_compress(const unsigned char *src, unsigned char *dst,
size_t *dst_len, void *private)
{
int ret = lzo1x_1_compress(src, PAGE_SIZE, dst, dst_len, private);
return ret == LZO_E_OK ? 0 : ret;
}
static int lzo_decompress(const unsigned char *src, size_t src_len,
unsigned char *dst)
{
size_t dst_len = PAGE_SIZE;
int ret = lzo1x_decompress_safe(src, src_len, dst, &dst_len);
return ret == LZO_E_OK ? 0 : ret;
}
struct zcomp_backend zcomp_lzo = {
.compress = lzo_compress,
.decompress = lzo_decompress,
.create = lzo_create,
.destroy = lzo_destroy,
.name = "lzo",
};

17
drivers/block/zram/zcomp_lzo.h Normal file
View File

@ -0,0 +1,17 @@
/*
* Copyright (C) 2014 Sergey Senozhatsky.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
#ifndef _ZCOMP_LZO_H_
#define _ZCOMP_LZO_H_
#include "zcomp.h"
extern struct zcomp_backend zcomp_lzo;
#endif /* _ZCOMP_LZO_H_ */

1162
drivers/block/zram/zram_drv.c Normal file
View File

File diff suppressed because it is too large Load Diff

85
drivers/staging/zram/zram_drv.h → drivers/block/zram/zram_drv.h
View File

@ -2,6 +2,7 @@
* Compressed RAM block device
*
* Copyright (C) 2008, 2009, 2010 Nitin Gupta
* 2012, 2013 Minchan Kim
*
* This code is released using a dual license strategy: BSD/GPL
* You can choose the licence that better fits your requirements.
@ -9,16 +10,15 @@
* Released under the terms of 3-clause BSD License
* Released under the terms of GNU General Public License Version 2.0
*
* Project home: http://compcache.googlecode.com
*/
#ifndef _ZRAM_DRV_H_
#define _ZRAM_DRV_H_
#include <linux/spinlock.h>
#include <linux/mutex.h>
#include <linux/zsmalloc.h>
#include "../zsmalloc/zsmalloc.h"
#include "zcomp.h"
/*
* Some arbitrary value. This is just to catch
@ -43,7 +43,6 @@ static const size_t max_zpage_size = PAGE_SIZE / 4 * 3;
/*-- End of configurable params */
#define SECTOR_SHIFT 9
#define SECTOR_SIZE (1 << SECTOR_SHIFT)
#define SECTORS_PER_PAGE_SHIFT (PAGE_SHIFT - SECTOR_SHIFT)
#define SECTORS_PER_PAGE (1 << SECTORS_PER_PAGE_SHIFT)
#define ZRAM_LOGICAL_BLOCK_SHIFT 12
@ -51,10 +50,24 @@ static const size_t max_zpage_size = PAGE_SIZE / 4 * 3;
#define ZRAM_SECTOR_PER_LOGICAL_BLOCK \
(1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT))
/* Flags for zram pages (table[page_no].flags) */
/*
* The lower ZRAM_FLAG_SHIFT bits of table.value is for
* object size (excluding header), the higher bits is for
* zram_pageflags.
*
* zram is mainly used for memory efficiency so we want to keep memory
* footprint small so we can squeeze size and flags into a field.
* The lower ZRAM_FLAG_SHIFT bits is for object size (excluding header),
* the higher bits is for zram_pageflags.
*/
#define ZRAM_FLAG_SHIFT 24
/* Flags for zram pages (table[page_no].value) */
enum zram_pageflags {
/* Page consists entirely of zeros */
ZRAM_ZERO,
ZRAM_ZERO = ZRAM_FLAG_SHIFT,
ZRAM_ACCESS, /* page is now accessed */
__NR_ZRAM_PAGEFLAGS,
};
@ -62,43 +75,35 @@ enum zram_pageflags {
/*-- Data structures */
/* Allocated for each disk page */
struct table {
struct zram_table_entry {
unsigned long handle;
u16 size; /* object size (excluding header) */
u8 count; /* object ref count (not yet used) */
u8 flags;
} __aligned(4);
unsigned long value;
};
struct zram_stats {
u64 compr_size; /* compressed size of pages stored */
u64 num_reads; /* failed + successful */
u64 num_writes; /* --do-- */
u64 failed_reads; /* should NEVER! happen */
u64 failed_writes; /* can happen when memory is too low */
u64 invalid_io; /* non-page-aligned I/O requests */
u64 notify_free; /* no. of swap slot free notifications */
u32 pages_zero; /* no. of zero filled pages */
u32 pages_stored; /* no. of pages currently stored */
u32 good_compress; /* % of pages with compression ratio<=50% */
u32 bad_compress; /* % of pages with compression ratio>=75% */
atomic64_t compr_data_size; /* compressed size of pages stored */
atomic64_t num_reads; /* failed + successful */
atomic64_t num_writes; /* --do-- */
atomic64_t failed_reads; /* can happen when memory is too low */
atomic64_t failed_writes; /* can happen when memory is too low */
atomic64_t invalid_io; /* non-page-aligned I/O requests */
atomic64_t notify_free; /* no. of swap slot free notifications */
atomic64_t zero_pages; /* no. of zero filled pages */
atomic64_t pages_stored; /* no. of pages currently stored */
atomic_long_t max_used_pages; /* no. of maximum pages stored */
};
struct zram_meta {
void *compress_workmem;
void *compress_buffer;
struct table *table;
struct zram_table_entry *table;
struct zs_pool *mem_pool;
};
struct zram {
struct zram_meta *meta;
spinlock_t stat64_lock; /* protect 64-bit stats */
struct rw_semaphore lock; /* protect compression buffers, table,
* 32bit stat counters against concurrent
* notifications, reads and writes */
struct request_queue *queue;
struct gendisk *disk;
int init_done;
struct zcomp *comp;
/* Prevent concurrent execution of device init, reset and R/W request */
struct rw_semaphore init_lock;
/*
@ -106,19 +111,13 @@ struct zram {
* we can store in a disk.
*/
u64 disksize; /* bytes */
int max_comp_streams;
struct zram_stats stats;
/*
* the number of pages zram can consume for storing compressed data
*/
unsigned long limit_pages;
char compressor[10];
};
extern struct zram *zram_devices;
unsigned int zram_get_num_devices(void);
#ifdef CONFIG_SYSFS
extern struct attribute_group zram_disk_attr_group;
#endif
extern void zram_reset_device(struct zram *zram);
extern struct zram_meta *zram_meta_alloc(u64 disksize);
extern void zram_meta_free(struct zram_meta *meta);
extern void zram_init_device(struct zram *zram, struct zram_meta *meta);
#endif

4
drivers/staging/Kconfig
View File

@ -72,10 +72,6 @@ source "drivers/staging/sep/Kconfig"
source "drivers/staging/iio/Kconfig"
source "drivers/staging/zsmalloc/Kconfig"
source "drivers/staging/zram/Kconfig"
source "drivers/staging/wlags49_h2/Kconfig"
source "drivers/staging/wlags49_h25/Kconfig"

2
drivers/staging/Makefile
View File

@ -30,8 +30,6 @@ obj-$(CONFIG_VT6656) += vt6656/
obj-$(CONFIG_VME_BUS) += vme/
obj-$(CONFIG_DX_SEP) += sep/
obj-$(CONFIG_IIO) += iio/
obj-$(CONFIG_ZRAM) += zram/
obj-$(CONFIG_ZSMALLOC) += zsmalloc/
obj-$(CONFIG_WLAGS49_H2) += wlags49_h2/
obj-$(CONFIG_WLAGS49_H25) += wlags49_h25/
obj-$(CONFIG_FB_SM7XX) += sm7xxfb/

3
drivers/staging/zram/Makefile
View File

@ -1,3 +0,0 @@
zram-y := zram_drv.o zram_sysfs.o
obj-$(CONFIG_ZRAM) += zram.o

77
drivers/staging/zram/zram.txt
View File

@ -1,77 +0,0 @@
zram: Compressed RAM based block devices
----------------------------------------
Project home: http://compcache.googlecode.com/
* Introduction
The zram module creates RAM based block devices named /dev/zram<id>
(<id> = 0, 1, ...). Pages written to these disks are compressed and stored
in memory itself. These disks allow very fast I/O and compression provides
good amounts of memory savings. Some of the usecases include /tmp storage,
use as swap disks, various caches under /var and maybe many more :)
Statistics for individual zram devices are exported through sysfs nodes at
/sys/block/zram<id>/
* Usage
Following shows a typical sequence of steps for using zram.
1) Load Module:
modprobe zram num_devices=4
This creates 4 devices: /dev/zram{0,1,2,3}
(num_devices parameter is optional. Default: 1)
2) Set Disksize
Set disk size by writing the value to sysfs node 'disksize'.
The value can be either in bytes or you can use mem suffixes.
Examples:
# Initialize /dev/zram0 with 50MB disksize
echo $((50*1024*1024)) > /sys/block/zram0/disksize
# Using mem suffixes
echo 256K > /sys/block/zram0/disksize
echo 512M > /sys/block/zram0/disksize
echo 1G > /sys/block/zram0/disksize
3) Activate:
mkswap /dev/zram0
swapon /dev/zram0
mkfs.ext4 /dev/zram1
mount /dev/zram1 /tmp
4) Stats:
Per-device statistics are exported as various nodes under
/sys/block/zram<id>/
disksize
num_reads
num_writes
invalid_io
notify_free
discard
zero_pages
orig_data_size
compr_data_size
mem_used_total
5) Deactivate:
swapoff /dev/zram0
umount /dev/zram1
6) Reset:
Write any positive value to 'reset' sysfs node
echo 1 > /sys/block/zram0/reset
echo 1 > /sys/block/zram1/reset
This frees all the memory allocated for the given device and
resets the disksize to zero. You must set the disksize again
before reusing the device.
Please report any problems at:
- Mailing list: linux-mm-cc at laptop dot org
- Issue tracker: http://code.google.com/p/compcache/issues/list
Nitin Gupta
ngupta@vflare.org

760
drivers/staging/zram/zram_drv.c
View File

@ -1,760 +0,0 @@
/*
* Compressed RAM block device
*
* Copyright (C) 2008, 2009, 2010 Nitin Gupta
*
* This code is released using a dual license strategy: BSD/GPL
* You can choose the licence that better fits your requirements.
*
* Released under the terms of 3-clause BSD License
* Released under the terms of GNU General Public License Version 2.0
*
* Project home: http://compcache.googlecode.com
*/
#define KMSG_COMPONENT "zram"
#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
#ifdef CONFIG_ZRAM_DEBUG
#define DEBUG
#endif
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/bio.h>
#include <linux/bitops.h>
#include <linux/blkdev.h>
#include <linux/buffer_head.h>
#include <linux/device.h>
#include <linux/genhd.h>
#include <linux/highmem.h>
#include <linux/slab.h>
#include <linux/lzo.h>
#include <linux/string.h>
#include <linux/vmalloc.h>
#include "zram_drv.h"
/* Globals */
static int zram_major;
struct zram *zram_devices;
/* Module params (documentation at end) */
static unsigned int num_devices = 1;
static void zram_stat64_add(struct zram *zram, u64 *v, u64 inc)
{
spin_lock(&zram->stat64_lock);
*v = *v + inc;
spin_unlock(&zram->stat64_lock);
}
static void zram_stat64_sub(struct zram *zram, u64 *v, u64 dec)
{
spin_lock(&zram->stat64_lock);
*v = *v - dec;
spin_unlock(&zram->stat64_lock);
}
static void zram_stat64_inc(struct zram *zram, u64 *v)
{
zram_stat64_add(zram, v, 1);
}
static int zram_test_flag(struct zram_meta *meta, u32 index,
enum zram_pageflags flag)
{
return meta->table[index].flags & BIT(flag);
}
static void zram_set_flag(struct zram_meta *meta, u32 index,
enum zram_pageflags flag)
{
meta->table[index].flags |= BIT(flag);
}
static void zram_clear_flag(struct zram_meta *meta, u32 index,
enum zram_pageflags flag)
{
meta->table[index].flags &= ~BIT(flag);
}
static int page_zero_filled(void *ptr)
{
unsigned int pos;
unsigned long *page;
page = (unsigned long *)ptr;
for (pos = 0; pos != PAGE_SIZE / sizeof(*page); pos++) {
if (page[pos])
return 0;
}
return 1;
}
static void zram_free_page(struct zram *zram, size_t index)
{
struct zram_meta *meta = zram->meta;
unsigned long handle = meta->table[index].handle;
u16 size = meta->table[index].size;
if (unlikely(!handle)) {
/*
* No memory is allocated for zero filled pages.
* Simply clear zero page flag.
*/
if (zram_test_flag(meta, index, ZRAM_ZERO)) {
zram_clear_flag(meta, index, ZRAM_ZERO);
zram->stats.pages_zero--;
}
return;
}
if (unlikely(size > max_zpage_size))
zram->stats.bad_compress--;
zs_free(meta->mem_pool, handle);
if (size <= PAGE_SIZE / 2)
zram->stats.good_compress--;
zram_stat64_sub(zram, &zram->stats.compr_size,
meta->table[index].size);
zram->stats.pages_stored--;
meta->table[index].handle = 0;
meta->table[index].size = 0;
}
static void handle_zero_page(struct bio_vec *bvec)
{
struct page *page = bvec->bv_page;
void *user_mem;
user_mem = kmap_atomic(page);
memset(user_mem + bvec->bv_offset, 0, bvec->bv_len);
kunmap_atomic(user_mem);
flush_dcache_page(page);
}
static inline int is_partial_io(struct bio_vec *bvec)
{
return bvec->bv_len != PAGE_SIZE;
}
static int zram_decompress_page(struct zram *zram, char *mem, u32 index)
{
int ret = LZO_E_OK;
size_t clen = PAGE_SIZE;
unsigned char *cmem;
struct zram_meta *meta = zram->meta;
unsigned long handle = meta->table[index].handle;
if (!handle || zram_test_flag(meta, index, ZRAM_ZERO)) {
memset(mem, 0, PAGE_SIZE);
return 0;
}
cmem = zs_map_object(meta->mem_pool, handle, ZS_MM_RO);
if (meta->table[index].size == PAGE_SIZE)
memcpy(mem, cmem, PAGE_SIZE);
else
ret = lzo1x_decompress_safe(cmem, meta->table[index].size,
mem, &clen);
zs_unmap_object(meta->mem_pool, handle);
/* Should NEVER happen. Return bio error if it does. */
if (unlikely(ret != LZO_E_OK)) {
pr_err("Decompression failed! err=%d, page=%u\n", ret, index);
zram_stat64_inc(zram, &zram->stats.failed_reads);
return ret;
}
return 0;
}
static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
u32 index, int offset, struct bio *bio)
{
int ret;
struct page *page;
unsigned char *user_mem, *uncmem = NULL;
struct zram_meta *meta = zram->meta;
page = bvec->bv_page;
if (unlikely(!meta->table[index].handle) ||
zram_test_flag(meta, index, ZRAM_ZERO)) {
handle_zero_page(bvec);
return 0;
}
if (is_partial_io(bvec))
/* Use a temporary buffer to decompress the page */
uncmem = kmalloc(PAGE_SIZE, GFP_NOIO);
user_mem = kmap_atomic(page);
if (!is_partial_io(bvec))
uncmem = user_mem;
if (!uncmem) {
pr_info("Unable to allocate temp memory\n");
ret = -ENOMEM;
goto out_cleanup;
}
ret = zram_decompress_page(zram, uncmem, index);
/* Should NEVER happen. Return bio error if it does. */
if (unlikely(ret != LZO_E_OK))
goto out_cleanup;
if (is_partial_io(bvec))
memcpy(user_mem + bvec->bv_offset, uncmem + offset,
bvec->bv_len);
flush_dcache_page(page);
ret = 0;
out_cleanup:
kunmap_atomic(user_mem);
if (is_partial_io(bvec))
kfree(uncmem);
return ret;
}
static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, u32 index,
int offset)
{
int ret = 0;
size_t clen;
unsigned long handle;
struct page *page;
unsigned char *user_mem, *cmem, *src, *uncmem = NULL;
struct zram_meta *meta = zram->meta;
page = bvec->bv_page;
src = meta->compress_buffer;
if (is_partial_io(bvec)) {
/*
* This is a partial IO. We need to read the full page
* before to write the changes.
*/
uncmem = kmalloc(PAGE_SIZE, GFP_NOIO);
if (!uncmem) {
ret = -ENOMEM;
goto out;
}
ret = zram_decompress_page(zram, uncmem, index);
if (ret)
goto out;
}
/*
* System overwrites unused sectors. Free memory associated
* with this sector now.
*/
if (meta->table[index].handle ||
zram_test_flag(meta, index, ZRAM_ZERO))
zram_free_page(zram, index);
user_mem = kmap_atomic(page);
if (is_partial_io(bvec)) {
memcpy(uncmem + offset, user_mem + bvec->bv_offset,
bvec->bv_len);
kunmap_atomic(user_mem);
user_mem = NULL;
} else {
uncmem = user_mem;
}
if (page_zero_filled(uncmem)) {
kunmap_atomic(user_mem);
zram->stats.pages_zero++;
zram_set_flag(meta, index, ZRAM_ZERO);
ret = 0;
goto out;
}
ret = lzo1x_1_compress(uncmem, PAGE_SIZE, src, &clen,
meta->compress_workmem);
if (!is_partial_io(bvec)) {
kunmap_atomic(user_mem);
user_mem = NULL;
uncmem = NULL;
}
if (unlikely(ret != LZO_E_OK)) {
pr_err("Compression failed! err=%d\n", ret);
goto out;
}
if (unlikely(clen > max_zpage_size)) {
zram->stats.bad_compress++;
clen = PAGE_SIZE;
src = NULL;
if (is_partial_io(bvec))
src = uncmem;
}
handle = zs_malloc(meta->mem_pool, clen);
if (!handle) {
pr_info("Error allocating memory for compressed "
"page: %u, size=%zu\n", index, clen);
ret = -ENOMEM;
goto out;
}
cmem = zs_map_object(meta->mem_pool, handle, ZS_MM_WO);
if ((clen == PAGE_SIZE) && !is_partial_io(bvec))
src = kmap_atomic(page);
memcpy(cmem, src, clen);
if ((clen == PAGE_SIZE) && !is_partial_io(bvec))
kunmap_atomic(src);
zs_unmap_object(meta->mem_pool, handle);
meta->table[index].handle = handle;
meta->table[index].size = clen;
/* Update stats */
zram_stat64_add(zram, &zram->stats.compr_size, clen);
zram->stats.pages_stored++;
if (clen <= PAGE_SIZE / 2)
zram->stats.good_compress++;
out:
if (is_partial_io(bvec))
kfree(uncmem);
if (ret)
zram_stat64_inc(zram, &zram->stats.failed_writes);
return ret;
}
static int zram_bvec_rw(struct zram *zram, struct bio_vec *bvec, u32 index,
int offset, struct bio *bio, int rw)
{
int ret;
if (rw == READ) {
down_read(&zram->lock);
ret = zram_bvec_read(zram, bvec, index, offset, bio);
up_read(&zram->lock);
} else {
down_write(&zram->lock);
ret = zram_bvec_write(zram, bvec, index, offset);
up_write(&zram->lock);
}
return ret;
}
static void update_position(u32 *index, int *offset, struct bio_vec *bvec)
{
if (*offset + bvec->bv_len >= PAGE_SIZE)
(*index)++;
*offset = (*offset + bvec->bv_len) % PAGE_SIZE;
}
static void __zram_make_request(struct zram *zram, struct bio *bio, int rw)
{
int i, offset;
u32 index;
struct bio_vec *bvec;
switch (rw) {
case READ:
zram_stat64_inc(zram, &zram->stats.num_reads);
break;
case WRITE:
zram_stat64_inc(zram, &zram->stats.num_writes);
break;
}
index = bio->bi_sector >> SECTORS_PER_PAGE_SHIFT;
offset = (bio->bi_sector & (SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT;
bio_for_each_segment(bvec, bio, i) {
int max_transfer_size = PAGE_SIZE - offset;
if (bvec->bv_len > max_transfer_size) {
/*
* zram_bvec_rw() can only make operation on a single
* zram page. Split the bio vector.
*/
struct bio_vec bv;
bv.bv_page = bvec->bv_page;
bv.bv_len = max_transfer_size;
bv.bv_offset = bvec->bv_offset;
if (zram_bvec_rw(zram, &bv, index, offset, bio, rw) < 0)
goto out;
bv.bv_len = bvec->bv_len - max_transfer_size;
bv.bv_offset += max_transfer_size;
if (zram_bvec_rw(zram, &bv, index+1, 0, bio, rw) < 0)
goto out;
} else
if (zram_bvec_rw(zram, bvec, index, offset, bio, rw)
< 0)
goto out;
update_position(&index, &offset, bvec);
}
set_bit(BIO_UPTODATE, &bio->bi_flags);
bio_endio(bio, 0);
return;
out:
bio_io_error(bio);
}
/*
* Check if request is within bounds and aligned on zram logical blocks.
*/
static inline int valid_io_request(struct zram *zram, struct bio *bio)
{
u64 start, end, bound;
/* unaligned request */
if (unlikely(bio->bi_sector & (ZRAM_SECTOR_PER_LOGICAL_BLOCK - 1)))
return 0;
if (unlikely(bio->bi_size & (ZRAM_LOGICAL_BLOCK_SIZE - 1)))
return 0;
start = bio->bi_sector;
end = start + (bio->bi_size >> SECTOR_SHIFT);
bound = zram->disksize >> SECTOR_SHIFT;
/* out of range range */
if (unlikely(start >= bound || end > bound || start > end))
return 0;
/* I/O request is valid */
return 1;
}
/*
* Handler function for all zram I/O requests.
*/
static void zram_make_request(struct request_queue *queue, struct bio *bio)
{
struct zram *zram = queue->queuedata;
down_read(&zram->init_lock);
if (unlikely(!zram->init_done))
goto error;
if (!valid_io_request(zram, bio)) {
zram_stat64_inc(zram, &zram->stats.invalid_io);
goto error;
}
__zram_make_request(zram, bio, bio_data_dir(bio));
up_read(&zram->init_lock);
return;
error:
up_read(&zram->init_lock);
bio_io_error(bio);
}
static void __zram_reset_device(struct zram *zram)
{
size_t index;
struct zram_meta *meta;
if (!zram->init_done)
return;
meta = zram->meta;
zram->init_done = 0;
/* Free all pages that are still in this zram device */
for (index = 0; index < zram->disksize >> PAGE_SHIFT; index++) {
unsigned long handle = meta->table[index].handle;
if (!handle)
continue;
zs_free(meta->mem_pool, handle);
}
zram_meta_free(zram->meta);
zram->meta = NULL;
/* Reset stats */
memset(&zram->stats, 0, sizeof(zram->stats));
zram->disksize = 0;
set_capacity(zram->disk, 0);
}
void zram_reset_device(struct zram *zram)
{
down_write(&zram->init_lock);
__zram_reset_device(zram);
up_write(&zram->init_lock);
}
void zram_meta_free(struct zram_meta *meta)
{
zs_destroy_pool(meta->mem_pool);
kfree(meta->compress_workmem);
free_pages((unsigned long)meta->compress_buffer, 1);
vfree(meta->table);
kfree(meta);
}
struct zram_meta *zram_meta_alloc(u64 disksize)
{
size_t num_pages;
struct zram_meta *meta = kmalloc(sizeof(*meta), GFP_KERNEL);
if (!meta)
goto out;
meta->compress_workmem = kzalloc(LZO1X_MEM_COMPRESS, GFP_KERNEL);
if (!meta->compress_workmem)
goto free_meta;
meta->compress_buffer =
(void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, 1);
if (!meta->compress_buffer) {
pr_err("Error allocating compressor buffer space\n");
goto free_workmem;
}
num_pages = disksize >> PAGE_SHIFT;
meta->table = vzalloc(num_pages * sizeof(*meta->table));
if (!meta->table) {
pr_err("Error allocating zram address table\n");
goto free_buffer;
}
meta->mem_pool = zs_create_pool(GFP_NOIO | __GFP_HIGHMEM);
if (!meta->mem_pool) {
pr_err("Error creating memory pool\n");
goto free_table;
}
return meta;
free_table:
vfree(meta->table);
free_buffer:
free_pages((unsigned long)meta->compress_buffer, 1);
free_workmem:
kfree(meta->compress_workmem);
free_meta:
kfree(meta);
meta = NULL;
out:
return meta;
}
void zram_init_device(struct zram *zram, struct zram_meta *meta)
{
if (zram->disksize > 2 * (totalram_pages << PAGE_SHIFT)) {
pr_info(
"There is little point creating a zram of greater than "
"twice the size of memory since we expect a 2:1 compression "
"ratio. Note that zram uses about 0.1%% of the size of "
"the disk when not in use so a huge zram is "
"wasteful.\n"
"\tMemory Size: %lu kB\n"
"\tSize you selected: %llu kB\n"
"Continuing anyway ...\n",
(totalram_pages << PAGE_SHIFT) >> 10, zram->disksize >> 10
);
}
/* zram devices sort of resembles non-rotational disks */
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, zram->disk->queue);
zram->meta = meta;
zram->init_done = 1;
pr_debug("Initialization done!\n");
}
static void zram_slot_free_notify(struct block_device *bdev,
unsigned long index)
{
struct zram *zram;
zram = bdev->bd_disk->private_data;
down_write(&zram->lock);
zram_free_page(zram, index);
up_write(&zram->lock);
zram_stat64_inc(zram, &zram->stats.notify_free);
}
static const struct block_device_operations zram_devops = {
.swap_slot_free_notify = zram_slot_free_notify,
.owner = THIS_MODULE
};
static int create_device(struct zram *zram, int device_id)
{
int ret = -ENOMEM;
init_rwsem(&zram->lock);
init_rwsem(&zram->init_lock);
spin_lock_init(&zram->stat64_lock);
zram->queue = blk_alloc_queue(GFP_KERNEL);
if (!zram->queue) {
pr_err("Error allocating disk queue for device %d\n",
device_id);
goto out;
}
blk_queue_make_request(zram->queue, zram_make_request);
zram->queue->queuedata = zram;
/* gendisk structure */
zram->disk = alloc_disk(1);
if (!zram->disk) {
pr_warn("Error allocating disk structure for device %d\n",
device_id);
goto out_free_queue;
}
zram->disk->major = zram_major;
zram->disk->first_minor = device_id;
zram->disk->fops = &zram_devops;
zram->disk->queue = zram->queue;
zram->disk->private_data = zram;
snprintf(zram->disk->disk_name, 16, "zram%d", device_id);
/* Actual capacity set using syfs (/sys/block/zram<id>/disksize */
set_capacity(zram->disk, 0);
/*
* To ensure that we always get PAGE_SIZE aligned
* and n*PAGE_SIZED sized I/O requests.
*/
blk_queue_physical_block_size(zram->disk->queue, PAGE_SIZE);
blk_queue_logical_block_size(zram->disk->queue,
ZRAM_LOGICAL_BLOCK_SIZE);
blk_queue_io_min(zram->disk->queue, PAGE_SIZE);
blk_queue_io_opt(zram->disk->queue, PAGE_SIZE);
add_disk(zram->disk);
ret = sysfs_create_group(&disk_to_dev(zram->disk)->kobj,
&zram_disk_attr_group);
if (ret < 0) {
pr_warn("Error creating sysfs group");
goto out_free_disk;
}
zram->init_done = 0;
return 0;
out_free_disk:
del_gendisk(zram->disk);
put_disk(zram->disk);
out_free_queue:
blk_cleanup_queue(zram->queue);
out:
return ret;
}
static void destroy_device(struct zram *zram)
{
sysfs_remove_group(&disk_to_dev(zram->disk)->kobj,
&zram_disk_attr_group);
if (zram->disk) {
del_gendisk(zram->disk);
put_disk(zram->disk);
}
if (zram->queue)
blk_cleanup_queue(zram->queue);
}
unsigned int zram_get_num_devices(void)
{
return num_devices;
}
static int __init zram_init(void)
{
int ret, dev_id;
if (num_devices > max_num_devices) {
pr_warn("Invalid value for num_devices: %u\n",
num_devices);
ret = -EINVAL;
goto out;
}
zram_major = register_blkdev(0, "zram");
if (zram_major <= 0) {
pr_warn("Unable to get major number\n");
ret = -EBUSY;
goto out;
}
/* Allocate the device array and initialize each one */
zram_devices = kzalloc(num_devices * sizeof(struct zram), GFP_KERNEL);
if (!zram_devices) {
ret = -ENOMEM;
goto unregister;
}
for (dev_id = 0; dev_id < num_devices; dev_id++) {
ret = create_device(&zram_devices[dev_id], dev_id);
if (ret)
goto free_devices;
}
pr_info("Created %u device(s) ...\n", num_devices);
return 0;
free_devices:
while (dev_id)
destroy_device(&zram_devices[--dev_id]);
kfree(zram_devices);
unregister:
unregister_blkdev(zram_major, "zram");
out:
return ret;
}
static void __exit zram_exit(void)
{
int i;
struct zram *zram;
for (i = 0; i < num_devices; i++) {
zram = &zram_devices[i];
get_disk(zram->disk);
destroy_device(zram);
zram_reset_device(zram);
put_disk(zram->disk);
}
unregister_blkdev(zram_major, "zram");
kfree(zram_devices);
pr_debug("Cleanup done!\n");
}
module_param(num_devices, uint, 0);
MODULE_PARM_DESC(num_devices, "Number of zram devices");
module_init(zram_init);
module_exit(zram_exit);
MODULE_LICENSE("Dual BSD/GPL");
MODULE_AUTHOR("Nitin Gupta <ngupta@vflare.org>");
MODULE_DESCRIPTION("Compressed RAM Block Device");

229
drivers/staging/zram/zram_sysfs.c
View File

@ -1,229 +0,0 @@
/*
* Compressed RAM block device
*
* Copyright (C) 2008, 2009, 2010 Nitin Gupta
*
* This code is released using a dual license strategy: BSD/GPL
* You can choose the licence that better fits your requirements.
*
* Released under the terms of 3-clause BSD License
* Released under the terms of GNU General Public License Version 2.0
*
* Project home: http://compcache.googlecode.com/
*/
#include <linux/device.h>
#include <linux/genhd.h>
#include <linux/mm.h>
#include <linux/kernel.h>
#include "zram_drv.h"
static u64 zram_stat64_read(struct zram *zram, u64 *v)
{
u64 val;
spin_lock(&zram->stat64_lock);
val = *v;
spin_unlock(&zram->stat64_lock);
return val;
}
static struct zram *dev_to_zram(struct device *dev)
{
int i;
struct zram *zram = NULL;
for (i = 0; i < zram_get_num_devices(); i++) {
zram = &zram_devices[i];
if (disk_to_dev(zram->disk) == dev)
break;
}
return zram;
}
static ssize_t disksize_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%llu\n", zram->disksize);
}
static ssize_t disksize_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
u64 disksize;
struct zram_meta *meta;
struct zram *zram = dev_to_zram(dev);
disksize = memparse(buf, NULL);
if (!disksize)
return -EINVAL;
disksize = PAGE_ALIGN(disksize);
meta = zram_meta_alloc(disksize);
down_write(&zram->init_lock);
if (zram->init_done) {
up_write(&zram->init_lock);
zram_meta_free(meta);
pr_info("Cannot change disksize for initialized device\n");
return -EBUSY;
}
zram->disksize = disksize;
set_capacity(zram->disk, zram->disksize >> SECTOR_SHIFT);
zram_init_device(zram, meta);
up_write(&zram->init_lock);
return len;
}
static ssize_t initstate_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%u\n", zram->init_done);
}
static ssize_t reset_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
int ret;
unsigned short do_reset;
struct zram *zram;
struct block_device *bdev;
zram = dev_to_zram(dev);
bdev = bdget_disk(zram->disk, 0);
/* Do not reset an active device! */
if (bdev->bd_holders)
return -EBUSY;
ret = kstrtou16(buf, 10, &do_reset);
if (ret)
return ret;
if (!do_reset)
return -EINVAL;
/* Make sure all pending I/O is finished */
if (bdev)
fsync_bdev(bdev);
zram_reset_device(zram);
return len;
}
static ssize_t num_reads_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%llu\n",
zram_stat64_read(zram, &zram->stats.num_reads));
}
static ssize_t num_writes_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%llu\n",
zram_stat64_read(zram, &zram->stats.num_writes));
}
static ssize_t invalid_io_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%llu\n",
zram_stat64_read(zram, &zram->stats.invalid_io));
}
static ssize_t notify_free_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%llu\n",
zram_stat64_read(zram, &zram->stats.notify_free));
}
static ssize_t zero_pages_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%u\n", zram->stats.pages_zero);
}
static ssize_t orig_data_size_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%llu\n",
(u64)(zram->stats.pages_stored) << PAGE_SHIFT);
}
static ssize_t compr_data_size_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct zram *zram = dev_to_zram(dev);
return sprintf(buf, "%llu\n",
zram_stat64_read(zram, &zram->stats.compr_size));
}
static ssize_t mem_used_total_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
u64 val = 0;
struct zram *zram = dev_to_zram(dev);
struct zram_meta *meta = zram->meta;
down_read(&zram->init_lock);
if (zram->init_done)
val = zs_get_total_size_bytes(meta->mem_pool);
up_read(&zram->init_lock);
return sprintf(buf, "%llu\n", val);
}
static DEVICE_ATTR(disksize, S_IRUGO | S_IWUSR,
disksize_show, disksize_store);
static DEVICE_ATTR(initstate, S_IRUGO, initstate_show, NULL);
static DEVICE_ATTR(reset, S_IWUSR, NULL, reset_store);
static DEVICE_ATTR(num_reads, S_IRUGO, num_reads_show, NULL);
static DEVICE_ATTR(num_writes, S_IRUGO, num_writes_show, NULL);
static DEVICE_ATTR(invalid_io, S_IRUGO, invalid_io_show, NULL);
static DEVICE_ATTR(notify_free, S_IRUGO, notify_free_show, NULL);
static DEVICE_ATTR(zero_pages, S_IRUGO, zero_pages_show, NULL);
static DEVICE_ATTR(orig_data_size, S_IRUGO, orig_data_size_show, NULL);
static DEVICE_ATTR(compr_data_size, S_IRUGO, compr_data_size_show, NULL);
static DEVICE_ATTR(mem_used_total, S_IRUGO, mem_used_total_show, NULL);
static struct attribute *zram_disk_attrs[] = {
&dev_attr_disksize.attr,
&dev_attr_initstate.attr,
&dev_attr_reset.attr,
&dev_attr_num_reads.attr,
&dev_attr_num_writes.attr,
&dev_attr_invalid_io.attr,
&dev_attr_notify_free.attr,
&dev_attr_zero_pages.attr,
&dev_attr_orig_data_size.attr,
&dev_attr_compr_data_size.attr,
&dev_attr_mem_used_total.attr,
NULL,
};
struct attribute_group zram_disk_attr_group = {
.attrs = zram_disk_attrs,
};

3
drivers/staging/zsmalloc/Makefile
View File

@ -1,3 +0,0 @@
zsmalloc-y := zsmalloc-main.o
obj-$(CONFIG_ZSMALLOC) += zsmalloc.o

47
include/linux/cpu.h
View File

@ -120,26 +120,46 @@ enum {
{ .notifier_call = fn, .priority = pri }; \
register_cpu_notifier(&fn##_nb); \
}
#define __cpu_notifier(fn, pri) { \
static struct notifier_block fn##_nb = \
{ .notifier_call = fn, .priority = pri }; \
__register_cpu_notifier(&fn##_nb); \
}
#else /* #if defined(CONFIG_HOTPLUG_CPU) || !defined(MODULE) */
#define cpu_notifier(fn, pri) do { (void)(fn); } while (0)
#define __cpu_notifier(fn, pri) do { (void)(fn); } while (0)
#endif /* #else #if defined(CONFIG_HOTPLUG_CPU) || !defined(MODULE) */
#ifdef CONFIG_HOTPLUG_CPU
extern int register_cpu_notifier(struct notifier_block *nb);
extern int __register_cpu_notifier(struct notifier_block *nb);
extern void unregister_cpu_notifier(struct notifier_block *nb);
extern void __unregister_cpu_notifier(struct notifier_block *nb);
#else
#ifndef MODULE
extern int register_cpu_notifier(struct notifier_block *nb);
extern int __register_cpu_notifier(struct notifier_block *nb);
#else
static inline int register_cpu_notifier(struct notifier_block *nb)
{
return 0;
}
static inline int __register_cpu_notifier(struct notifier_block *nb)
{
return 0;
}
#endif
static inline void unregister_cpu_notifier(struct notifier_block *nb)
{
}
static inline void __unregister_cpu_notifier(struct notifier_block *nb)
{
}
#endif
int cpu_up(unsigned int cpu);
@ -147,19 +167,32 @@ void notify_cpu_starting(unsigned int cpu);
extern void cpu_maps_update_begin(void);
extern void cpu_maps_update_done(void);
#define cpu_notifier_register_begin cpu_maps_update_begin
#define cpu_notifier_register_done cpu_maps_update_done
#else /* CONFIG_SMP */
#define cpu_notifier(fn, pri) do { (void)(fn); } while (0)
#define __cpu_notifier(fn, pri) do { (void)(fn); } while (0)
static inline int register_cpu_notifier(struct notifier_block *nb)
{
return 0;
}
static inline int __register_cpu_notifier(struct notifier_block *nb)
{
return 0;
}
static inline void unregister_cpu_notifier(struct notifier_block *nb)
{
}
static inline void __unregister_cpu_notifier(struct notifier_block *nb)
{
}
static inline void cpu_maps_update_begin(void)
{
}
@ -168,6 +201,14 @@ static inline void cpu_maps_update_done(void)
{
}
static inline void cpu_notifier_register_begin(void)
{
}
static inline void cpu_notifier_register_done(void)
{
}
#endif /* CONFIG_SMP */
extern struct bus_type cpu_subsys;
@ -179,8 +220,11 @@ extern void put_online_cpus(void);
extern void cpu_hotplug_disable(void);
extern void cpu_hotplug_enable(void);
#define hotcpu_notifier(fn, pri) cpu_notifier(fn, pri)
#define __hotcpu_notifier(fn, pri) __cpu_notifier(fn, pri)
#define register_hotcpu_notifier(nb) register_cpu_notifier(nb)
#define __register_hotcpu_notifier(nb) __register_cpu_notifier(nb)
#define unregister_hotcpu_notifier(nb) unregister_cpu_notifier(nb)
#define __unregister_hotcpu_notifier(nb) __unregister_cpu_notifier(nb)
void clear_tasks_mm_cpumask(int cpu);
int cpu_down(unsigned int cpu);
@ -204,9 +248,12 @@ static inline void cpu_hotplug_driver_unlock(void)
#define cpu_hotplug_disable() do { } while (0)
#define cpu_hotplug_enable() do { } while (0)
#define hotcpu_notifier(fn, pri) do { (void)(fn); } while (0)
#define __hotcpu_notifier(fn, pri) do { (void)(fn); } while (0)
/* These aren't inline functions due to a GCC bug. */
#define register_hotcpu_notifier(nb) ({ (void)(nb); 0; })
#define __register_hotcpu_notifier(nb) ({ (void)(nb); 0; })
#define unregister_hotcpu_notifier(nb) ({ (void)(nb); })
#define __unregister_hotcpu_notifier(nb) ({ (void)(nb); })
#endif /* CONFIG_HOTPLUG_CPU */
#ifdef CONFIG_PM_SLEEP_SMP

87
include/linux/lz4.h Normal file
View File

@ -0,0 +1,87 @@
#ifndef __LZ4_H__
#define __LZ4_H__
/*
* LZ4 Kernel Interface
*
* Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#define LZ4_MEM_COMPRESS (4096 * sizeof(unsigned char *))
#define LZ4HC_MEM_COMPRESS (65538 * sizeof(unsigned char *))
/*
* lz4_compressbound()
* Provides the maximum size that LZ4 may output in a "worst case" scenario
* (input data not compressible)
*/
static inline size_t lz4_compressbound(size_t isize)
{
return isize + (isize / 255) + 16;
}
/*
* lz4_compress()
* src : source address of the original data
* src_len : size of the original data
* dst : output buffer address of the compressed data
* This requires 'dst' of size LZ4_COMPRESSBOUND.
* dst_len : is the output size, which is returned after compress done
* workmem : address of the working memory.
* This requires 'workmem' of size LZ4_MEM_COMPRESS.
* return : Success if return 0
* Error if return (< 0)
* note : Destination buffer and workmem must be already allocated with
* the defined size.
*/
int lz4_compress(const unsigned char *src, size_t src_len,
unsigned char *dst, size_t *dst_len, void *wrkmem);
/*
* lz4hc_compress()
* src : source address of the original data
* src_len : size of the original data
* dst : output buffer address of the compressed data
* This requires 'dst' of size LZ4_COMPRESSBOUND.
* dst_len : is the output size, which is returned after compress done
* workmem : address of the working memory.
* This requires 'workmem' of size LZ4HC_MEM_COMPRESS.
* return : Success if return 0
* Error if return (< 0)
* note : Destination buffer and workmem must be already allocated with
* the defined size.
*/
int lz4hc_compress(const unsigned char *src, size_t src_len,
unsigned char *dst, size_t *dst_len, void *wrkmem);
/*
* lz4_decompress()
* src : source address of the compressed data
* src_len : is the input size, whcih is returned after decompress done
* dest : output buffer address of the decompressed data
* actual_dest_len: is the size of uncompressed data, supposing it's known
* return : Success if return 0
* Error if return (< 0)
* note : Destination buffer must be already allocated.
* slightly faster than lz4_decompress_unknownoutputsize()
*/
int lz4_decompress(const unsigned char *src, size_t *src_len,
unsigned char *dest, size_t actual_dest_len);
/*
* lz4_decompress_unknownoutputsize()
* src : source address of the compressed data
* src_len : is the input size, therefore the compressed size
* dest : output buffer address of the decompressed data
* dest_len: is the max size of the destination buffer, which is
* returned with actual size of decompressed data after
* decompress done
* return : Success if return 0
* Error if return (< 0)
* note : Destination buffer must be already allocated.
*/
int lz4_decompress_unknownoutputsize(const unsigned char *src, size_t src_len,
unsigned char *dest, size_t *dest_len);
#endif

106
include/linux/zpool.h Normal file
View File

@ -0,0 +1,106 @@
/*
* zpool memory storage api
*
* Copyright (C) 2014 Dan Streetman
*
* This is a common frontend for the zbud and zsmalloc memory
* storage pool implementations. Typically, this is used to
* store compressed memory.
*/
#ifndef _ZPOOL_H_
#define _ZPOOL_H_
struct zpool;
struct zpool_ops {
int (*evict)(struct zpool *pool, unsigned long handle);
};
/*
* Control how a handle is mapped. It will be ignored if the
* implementation does not support it. Its use is optional.
* Note that this does not refer to memory protection, it
* refers to how the memory will be copied in/out if copying
* is necessary during mapping; read-write is the safest as
* it copies the existing memory in on map, and copies the
* changed memory back out on unmap. Write-only does not copy
* in the memory and should only be used for initialization.
* If in doubt, use ZPOOL_MM_DEFAULT which is read-write.
*/
enum zpool_mapmode {
ZPOOL_MM_RW, /* normal read-write mapping */
ZPOOL_MM_RO, /* read-only (no copy-out at unmap time) */
ZPOOL_MM_WO, /* write-only (no copy-in at map time) */
ZPOOL_MM_DEFAULT = ZPOOL_MM_RW
};
struct zpool *zpool_create_pool(char *type, gfp_t gfp, struct zpool_ops *ops);
char *zpool_get_type(struct zpool *pool);
void zpool_destroy_pool(struct zpool *pool);
int zpool_malloc(struct zpool *pool, size_t size, gfp_t gfp,
unsigned long *handle);
void zpool_free(struct zpool *pool, unsigned long handle);
int zpool_shrink(struct zpool *pool, unsigned int pages,
unsigned int *reclaimed);
void *zpool_map_handle(struct zpool *pool, unsigned long handle,
enum zpool_mapmode mm);
void zpool_unmap_handle(struct zpool *pool, unsigned long handle);
u64 zpool_get_total_size(struct zpool *pool);
/**
* struct zpool_driver - driver implementation for zpool
* @type: name of the driver.
* @list: entry in the list of zpool drivers.
* @create: create a new pool.
* @destroy: destroy a pool.
* @malloc: allocate mem from a pool.
* @free: free mem from a pool.
* @shrink: shrink the pool.
* @map: map a handle.
* @unmap: unmap a handle.
* @total_size: get total size of a pool.
*
* This is created by a zpool implementation and registered
* with zpool.
*/
struct zpool_driver {
char *type;
struct module *owner;
atomic_t refcount;
struct list_head list;
void *(*create)(gfp_t gfp, struct zpool_ops *ops);
void (*destroy)(void *pool);
int (*malloc)(void *pool, size_t size, gfp_t gfp,
unsigned long *handle);
void (*free)(void *pool, unsigned long handle);
int (*shrink)(void *pool, unsigned int pages,
unsigned int *reclaimed);
void *(*map)(void *pool, unsigned long handle,
enum zpool_mapmode mm);
void (*unmap)(void *pool, unsigned long handle);
u64 (*total_size)(void *pool);
};
void zpool_register_driver(struct zpool_driver *driver);
int zpool_unregister_driver(struct zpool_driver *driver);
int zpool_evict(void *pool, unsigned long handle);
#endif

3
drivers/staging/zsmalloc/zsmalloc.h → include/linux/zsmalloc.h
View File

@ -2,6 +2,7 @@
* zsmalloc memory allocator
*
* Copyright (C) 2011 Nitin Gupta
* Copyright (C) 2012, 2013 Minchan Kim
*
* This code is released using a dual license strategy: BSD/GPL
* You can choose the license that better fits your requirements.
@ -38,6 +39,6 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
enum zs_mapmode mm);
void zs_unmap_object(struct zs_pool *pool, unsigned long handle);
u64 zs_get_total_size_bytes(struct zs_pool *pool);
unsigned long zs_get_total_pages(struct zs_pool *pool);
#endif

21
kernel/cpu.c
View File

@ -27,18 +27,23 @@
static DEFINE_MUTEX(cpu_add_remove_lock);
/*
* The following two API's must be used when attempting
* to serialize the updates to cpu_online_mask, cpu_present_mask.
* The following two APIs (cpu_maps_update_begin/done) must be used when
* attempting to serialize the updates to cpu_online_mask & cpu_present_mask.
* The APIs cpu_notifier_register_begin/done() must be used to protect CPU
* hotplug callback (un)registration performed using __register_cpu_notifier()
* or __unregister_cpu_notifier().
*/
void cpu_maps_update_begin(void)
{
mutex_lock(&cpu_add_remove_lock);
}
EXPORT_SYMBOL(cpu_notifier_register_begin);
void cpu_maps_update_done(void)
{
mutex_unlock(&cpu_add_remove_lock);
}
EXPORT_SYMBOL(cpu_notifier_register_done);
static RAW_NOTIFIER_HEAD(cpu_chain);
@ -169,6 +174,11 @@ int __ref register_cpu_notifier(struct notifier_block *nb)
return ret;
}
int __ref __register_cpu_notifier(struct notifier_block *nb)
{
return raw_notifier_chain_register(&cpu_chain, nb);
}
static int __cpu_notify(unsigned long val, void *v, int nr_to_call,
int *nr_calls)
{
@ -192,6 +202,7 @@ static void cpu_notify_nofail(unsigned long val, void *v)
BUG_ON(cpu_notify(val, v));
}
EXPORT_SYMBOL(register_cpu_notifier);
EXPORT_SYMBOL(__register_cpu_notifier);
void __ref unregister_cpu_notifier(struct notifier_block *nb)
{
@ -201,6 +212,12 @@ void __ref unregister_cpu_notifier(struct notifier_block *nb)
}
EXPORT_SYMBOL(unregister_cpu_notifier);
void __ref __unregister_cpu_notifier(struct notifier_block *nb)
{
raw_notifier_chain_unregister(&cpu_chain, nb);
}
EXPORT_SYMBOL(__unregister_cpu_notifier);
/**
* clear_tasks_mm_cpumask - Safely clear tasks' mm_cpumask for a CPU
* @cpu: a CPU id

9
lib/Kconfig
View File

@ -189,6 +189,15 @@ config LZO_COMPRESS
config LZO_DECOMPRESS
tristate
config LZ4_COMPRESS
tristate
config LZ4HC_COMPRESS
tristate
config LZ4_DECOMPRESS
tristate
source "lib/xz/Kconfig"
#

3
lib/Makefile
View File

@ -76,6 +76,9 @@ obj-$(CONFIG_REED_SOLOMON) += reed_solomon/
obj-$(CONFIG_BCH) += bch.o
obj-$(CONFIG_LZO_COMPRESS) += lzo/
obj-$(CONFIG_LZO_DECOMPRESS) += lzo/
obj-$(CONFIG_LZ4_COMPRESS) += lz4/
obj-$(CONFIG_LZ4HC_COMPRESS) += lz4/
obj-$(CONFIG_LZ4_DECOMPRESS) += lz4/
obj-$(CONFIG_XZ_DEC) += xz/
obj-$(CONFIG_RAID6_PQ) += raid6/

3
lib/lz4/Makefile Normal file
View File

@ -0,0 +1,3 @@
obj-$(CONFIG_LZ4_COMPRESS) += lz4_compress.o
obj-$(CONFIG_LZ4HC_COMPRESS) += lz4hc_compress.o
obj-$(CONFIG_LZ4_DECOMPRESS) += lz4_decompress.o

443
lib/lz4/lz4_compress.c Normal file
View File

@ -0,0 +1,443 @@
/*
* LZ4 - Fast LZ compression algorithm
* Copyright (C) 2011-2012, Yann Collet.
* BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* You can contact the author at :
* - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
* - LZ4 source repository : http://code.google.com/p/lz4/
*
* Changed for kernel use by:
* Chanho Min <chanho.min@lge.com>
*/
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/lz4.h>
#include <asm/unaligned.h>
#include "lz4defs.h"
/*
* LZ4_compressCtx :
* -----------------
* Compress 'isize' bytes from 'source' into an output buffer 'dest' of
* maximum size 'maxOutputSize'. * If it cannot achieve it, compression
* will stop, and result of the function will be zero.
* return : the number of bytes written in buffer 'dest', or 0 if the
* compression fails
*/
static inline int lz4_compressctx(void *ctx,
const char *source,
char *dest,
int isize,
int maxoutputsize)
{
HTYPE *hashtable = (HTYPE *)ctx;
const u8 *ip = (u8 *)source;
#if LZ4_ARCH64
const BYTE * const base = ip;
#else
const int base = 0;
#endif
const u8 *anchor = ip;
const u8 *const iend = ip + isize;
const u8 *const mflimit = iend - MFLIMIT;
#define MATCHLIMIT (iend - LASTLITERALS)
u8 *op = (u8 *) dest;
u8 *const oend = op + maxoutputsize;
int length;
const int skipstrength = SKIPSTRENGTH;
u32 forwardh;
int lastrun;
/* Init */
if (isize < MINLENGTH)
goto _last_literals;
memset((void *)hashtable, 0, LZ4_MEM_COMPRESS);
/* First Byte */
hashtable[LZ4_HASH_VALUE(ip)] = ip - base;
ip++;
forwardh = LZ4_HASH_VALUE(ip);
/* Main Loop */
for (;;) {
int findmatchattempts = (1U << skipstrength) + 3;
const u8 *forwardip = ip;
const u8 *ref;
u8 *token;
/* Find a match */
do {
u32 h = forwardh;
int step = findmatchattempts++ >> skipstrength;
ip = forwardip;
forwardip = ip + step;
if (unlikely(forwardip > mflimit))
goto _last_literals;
forwardh = LZ4_HASH_VALUE(forwardip);
ref = base + hashtable[h];
hashtable[h] = ip - base;
} while ((ref < ip - MAX_DISTANCE) || (A32(ref) != A32(ip)));
/* Catch up */
while ((ip > anchor) && (ref > (u8 *)source) &&
unlikely(ip[-1] == ref[-1])) {
ip--;
ref--;
}
/* Encode Literal length */
length = (int)(ip - anchor);
token = op++;
/* check output limit */
if (unlikely(op + length + (2 + 1 + LASTLITERALS) +
(length >> 8) > oend))
return 0;
if (length >= (int)RUN_MASK) {
int len;
*token = (RUN_MASK << ML_BITS);
len = length - RUN_MASK;
for (; len > 254 ; len -= 255)
*op++ = 255;
*op++ = (u8)len;
} else
*token = (length << ML_BITS);
/* Copy Literals */
LZ4_BLINDCOPY(anchor, op, length);
_next_match:
/* Encode Offset */
LZ4_WRITE_LITTLEENDIAN_16(op, (u16)(ip - ref));
/* Start Counting */
ip += MINMATCH;
/* MinMatch verified */
ref += MINMATCH;
anchor = ip;
while (likely(ip < MATCHLIMIT - (STEPSIZE - 1))) {
#if LZ4_ARCH64
u64 diff = A64(ref) ^ A64(ip);
#else
u32 diff = A32(ref) ^ A32(ip);
#endif
if (!diff) {
ip += STEPSIZE;
ref += STEPSIZE;
continue;
}
ip += LZ4_NBCOMMONBYTES(diff);
goto _endcount;
}
#if LZ4_ARCH64
if ((ip < (MATCHLIMIT - 3)) && (A32(ref) == A32(ip))) {
ip += 4;
ref += 4;
}
#endif
if ((ip < (MATCHLIMIT - 1)) && (A16(ref) == A16(ip))) {
ip += 2;
ref += 2;
}
if ((ip < MATCHLIMIT) && (*ref == *ip))
ip++;
_endcount:
/* Encode MatchLength */
length = (int)(ip - anchor);
/* Check output limit */
if (unlikely(op + (1 + LASTLITERALS) + (length >> 8) > oend))
return 0;
if (length >= (int)ML_MASK) {
*token += ML_MASK;
length -= ML_MASK;
for (; length > 509 ; length -= 510) {
*op++ = 255;
*op++ = 255;
}
if (length > 254) {
length -= 255;
*op++ = 255;
}
*op++ = (u8)length;
} else
*token += length;
/* Test end of chunk */
if (ip > mflimit) {
anchor = ip;
break;
}
/* Fill table */
hashtable[LZ4_HASH_VALUE(ip-2)] = ip - 2 - base;
/* Test next position */
ref = base + hashtable[LZ4_HASH_VALUE(ip)];
hashtable[LZ4_HASH_VALUE(ip)] = ip - base;
if ((ref > ip - (MAX_DISTANCE + 1)) && (A32(ref) == A32(ip))) {
token = op++;
*token = 0;
goto _next_match;
}
/* Prepare next loop */
anchor = ip++;
forwardh = LZ4_HASH_VALUE(ip);
}
_last_literals:
/* Encode Last Literals */
lastrun = (int)(iend - anchor);
if (((char *)op - dest) + lastrun + 1
+ ((lastrun + 255 - RUN_MASK) / 255) > (u32)maxoutputsize)
return 0;
if (lastrun >= (int)RUN_MASK) {
*op++ = (RUN_MASK << ML_BITS);
lastrun -= RUN_MASK;
for (; lastrun > 254 ; lastrun -= 255)
*op++ = 255;
*op++ = (u8)lastrun;
} else
*op++ = (lastrun << ML_BITS);
memcpy(op, anchor, iend - anchor);
op += iend - anchor;
/* End */
return (int)(((char *)op) - dest);
}
static inline int lz4_compress64kctx(void *ctx,
const char *source,
char *dest,
int isize,
int maxoutputsize)
{
u16 *hashtable = (u16 *)ctx;
const u8 *ip = (u8 *) source;
const u8 *anchor = ip;
const u8 *const base = ip;
const u8 *const iend = ip + isize;
const u8 *const mflimit = iend - MFLIMIT;
#define MATCHLIMIT (iend - LASTLITERALS)
u8 *op = (u8 *) dest;
u8 *const oend = op + maxoutputsize;
int len, length;
const int skipstrength = SKIPSTRENGTH;
u32 forwardh;
int lastrun;
/* Init */
if (isize < MINLENGTH)
goto _last_literals;
memset((void *)hashtable, 0, LZ4_MEM_COMPRESS);
/* First Byte */
ip++;
forwardh = LZ4_HASH64K_VALUE(ip);
/* Main Loop */
for (;;) {
int findmatchattempts = (1U << skipstrength) + 3;
const u8 *forwardip = ip;
const u8 *ref;
u8 *token;
/* Find a match */
do {
u32 h = forwardh;
int step = findmatchattempts++ >> skipstrength;
ip = forwardip;
forwardip = ip + step;
if (forwardip > mflimit)
goto _last_literals;
forwardh = LZ4_HASH64K_VALUE(forwardip);
ref = base + hashtable[h];
hashtable[h] = (u16)(ip - base);
} while (A32(ref) != A32(ip));
/* Catch up */
while ((ip > anchor) && (ref > (u8 *)source)
&& (ip[-1] == ref[-1])) {
ip--;
ref--;
}
/* Encode Literal length */
length = (int)(ip - anchor);
token = op++;
/* Check output limit */
if (unlikely(op + length + (2 + 1 + LASTLITERALS)
+ (length >> 8) > oend))
return 0;
if (length >= (int)RUN_MASK) {
*token = (RUN_MASK << ML_BITS);
len = length - RUN_MASK;
for (; len > 254 ; len -= 255)
*op++ = 255;
*op++ = (u8)len;
} else
*token = (length << ML_BITS);
/* Copy Literals */
LZ4_BLINDCOPY(anchor, op, length);
_next_match:
/* Encode Offset */
LZ4_WRITE_LITTLEENDIAN_16(op, (u16)(ip - ref));
/* Start Counting */
ip += MINMATCH;
/* MinMatch verified */
ref += MINMATCH;
anchor = ip;
while (ip < MATCHLIMIT - (STEPSIZE - 1)) {
#if LZ4_ARCH64
u64 diff = A64(ref) ^ A64(ip);
#else
u32 diff = A32(ref) ^ A32(ip);
#endif
if (!diff) {
ip += STEPSIZE;
ref += STEPSIZE;
continue;
}
ip += LZ4_NBCOMMONBYTES(diff);
goto _endcount;
}
#if LZ4_ARCH64
if ((ip < (MATCHLIMIT - 3)) && (A32(ref) == A32(ip))) {
ip += 4;
ref += 4;
}
#endif
if ((ip < (MATCHLIMIT - 1)) && (A16(ref) == A16(ip))) {
ip += 2;
ref += 2;
}
if ((ip < MATCHLIMIT) && (*ref == *ip))
ip++;
_endcount:
/* Encode MatchLength */
len = (int)(ip - anchor);
/* Check output limit */
if (unlikely(op + (1 + LASTLITERALS) + (len >> 8) > oend))
return 0;
if (len >= (int)ML_MASK) {
*token += ML_MASK;
len -= ML_MASK;
for (; len > 509 ; len -= 510) {
*op++ = 255;
*op++ = 255;
}
if (len > 254) {
len -= 255;
*op++ = 255;
}
*op++ = (u8)len;
} else
*token += len;
/* Test end of chunk */
if (ip > mflimit) {
anchor = ip;
break;
}
/* Fill table */
hashtable[LZ4_HASH64K_VALUE(ip-2)] = (u16)(ip - 2 - base);
/* Test next position */
ref = base + hashtable[LZ4_HASH64K_VALUE(ip)];
hashtable[LZ4_HASH64K_VALUE(ip)] = (u16)(ip - base);
if (A32(ref) == A32(ip)) {
token = op++;
*token = 0;
goto _next_match;
}
/* Prepare next loop */
anchor = ip++;
forwardh = LZ4_HASH64K_VALUE(ip);
}
_last_literals:
/* Encode Last Literals */
lastrun = (int)(iend - anchor);
if (op + lastrun + 1 + (lastrun - RUN_MASK + 255) / 255 > oend)
return 0;
if (lastrun >= (int)RUN_MASK) {
*op++ = (RUN_MASK << ML_BITS);
lastrun -= RUN_MASK;
for (; lastrun > 254 ; lastrun -= 255)
*op++ = 255;
*op++ = (u8)lastrun;
} else
*op++ = (lastrun << ML_BITS);
memcpy(op, anchor, iend - anchor);
op += iend - anchor;
/* End */
return (int)(((char *)op) - dest);
}
int lz4_compress(const unsigned char *src, size_t src_len,
unsigned char *dst, size_t *dst_len, void *wrkmem)
{
int ret = -1;
int out_len = 0;
if (src_len < LZ4_64KLIMIT)
out_len = lz4_compress64kctx(wrkmem, src, dst, src_len,
lz4_compressbound(src_len));
else
out_len = lz4_compressctx(wrkmem, src, dst, src_len,
lz4_compressbound(src_len));
if (out_len < 0)
goto exit;
*dst_len = out_len;
return 0;
exit:
return ret;
}
EXPORT_SYMBOL_GPL(lz4_compress);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("LZ4 compressor");

326
lib/lz4/lz4_decompress.c Normal file
View File

@ -0,0 +1,326 @@
/*
* LZ4 Decompressor for Linux kernel
*
* Copyright (C) 2013 LG Electronics Co., Ltd. (http://www.lge.com/)
*
* Based on LZ4 implementation by Yann Collet.
*
* LZ4 - Fast LZ compression algorithm
* Copyright (C) 2011-2012, Yann Collet.
* BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* You can contact the author at :
* - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
* - LZ4 source repository : http://code.google.com/p/lz4/
*/
#ifndef STATIC
#include <linux/module.h>
#include <linux/kernel.h>
#endif
#include <linux/lz4.h>
#include <asm/unaligned.h>
#include "lz4defs.h"
static int lz4_uncompress(const char *source, char *dest, int osize)
{
const BYTE *ip = (const BYTE *) source;
const BYTE *ref;
BYTE *op = (BYTE *) dest;
BYTE * const oend = op + osize;
BYTE *cpy;
unsigned token;
size_t length;
size_t dec32table[] = {0, 3, 2, 3, 0, 0, 0, 0};
#if LZ4_ARCH64
size_t dec64table[] = {0, 0, 0, -1, 0, 1, 2, 3};
#endif
while (1) {
/* get runlength */
token = *ip++;
length = (token >> ML_BITS);
if (length == RUN_MASK) {
size_t len;
len = *ip++;
for (; len == 255; length += 255)
len = *ip++;
length += len;
}
/* copy literals */
cpy = op + length;
if (unlikely(cpy > oend - COPYLENGTH)) {
/*
* Error: not enough place for another match
* (min 4) + 5 literals
*/
if (cpy != oend)
goto _output_error;
memcpy(op, ip, length);
ip += length;
break; /* EOF */
}
LZ4_WILDCOPY(ip, op, cpy);
ip -= (op - cpy);
op = cpy;
/* get offset */
LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip);
ip += 2;
/* Error: offset create reference outside destination buffer */
if (unlikely(ref < (BYTE *const) dest))
goto _output_error;
/* get matchlength */
length = token & ML_MASK;
if (length == ML_MASK) {
for (; *ip == 255; length += 255)
ip++;
length += *ip++;
}
/* copy repeated sequence */
if (unlikely((op - ref) < STEPSIZE)) {
#if LZ4_ARCH64
size_t dec64 = dec64table[op - ref];
#else
const int dec64 = 0;
#endif
op[0] = ref[0];
op[1] = ref[1];
op[2] = ref[2];
op[3] = ref[3];
op += 4;
ref += 4;
ref -= dec32table[op-ref];
PUT4(ref, op);
op += STEPSIZE - 4;
ref -= dec64;
} else {
LZ4_COPYSTEP(ref, op);
}
cpy = op + length - (STEPSIZE - 4);
if (cpy > (oend - COPYLENGTH)) {
/* Error: request to write beyond destination buffer */
if (cpy > oend)
goto _output_error;
LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
while (op < cpy)
*op++ = *ref++;
op = cpy;
/*
* Check EOF (should never happen, since last 5 bytes
* are supposed to be literals)
*/
if (op == oend)
goto _output_error;
continue;
}
LZ4_SECURECOPY(ref, op, cpy);
op = cpy; /* correction */
}
/* end of decoding */
return (int) (((char *)ip) - source);
/* write overflow error detected */
_output_error:
return (int) (-(((char *)ip) - source));
}
static int lz4_uncompress_unknownoutputsize(const char *source, char *dest,
int isize, size_t maxoutputsize)
{
const BYTE *ip = (const BYTE *) source;
const BYTE *const iend = ip + isize;
const BYTE *ref;
BYTE *op = (BYTE *) dest;
BYTE * const oend = op + maxoutputsize;
BYTE *cpy;
size_t dec32table[] = {0, 3, 2, 3, 0, 0, 0, 0};
#if LZ4_ARCH64
size_t dec64table[] = {0, 0, 0, -1, 0, 1, 2, 3};
#endif
/* Main Loop */
while (ip < iend) {
unsigned token;
size_t length;
/* get runlength */
token = *ip++;
length = (token >> ML_BITS);
if (length == RUN_MASK) {
int s = 255;
while ((ip < iend) && (s == 255)) {
s = *ip++;
length += s;
}
}
/* copy literals */
cpy = op + length;
if ((cpy > oend - COPYLENGTH) ||
(ip + length > iend - COPYLENGTH)) {
if (cpy > oend)
goto _output_error;/* writes beyond buffer */
if (ip + length != iend)
goto _output_error;/*
* Error: LZ4 format requires
* to consume all input
* at this stage
*/
memcpy(op, ip, length);
op += length;
break;/* Necessarily EOF, due to parsing restrictions */
}
LZ4_WILDCOPY(ip, op, cpy);
ip -= (op - cpy);
op = cpy;
/* get offset */
LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip);
ip += 2;
if (ref < (BYTE * const) dest)
goto _output_error;
/*
* Error : offset creates reference
* outside of destination buffer
*/
/* get matchlength */
length = (token & ML_MASK);
if (length == ML_MASK) {
while (ip < iend) {
int s = *ip++;
length += s;
if (s == 255)
continue;
break;
}
}
/* copy repeated sequence */
if (unlikely((op - ref) < STEPSIZE)) {
#if LZ4_ARCH64
size_t dec64 = dec64table[op - ref];
#else
const int dec64 = 0;
#endif
op[0] = ref[0];
op[1] = ref[1];
op[2] = ref[2];
op[3] = ref[3];
op += 4;
ref += 4;
ref -= dec32table[op - ref];
PUT4(ref, op);
op += STEPSIZE - 4;
ref -= dec64;
} else {
LZ4_COPYSTEP(ref, op);
}
cpy = op + length - (STEPSIZE-4);
if (cpy > oend - COPYLENGTH) {
if (cpy > oend)
goto _output_error; /* write outside of buf */
LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
while (op < cpy)
*op++ = *ref++;
op = cpy;
/*
* Check EOF (should never happen, since last 5 bytes
* are supposed to be literals)
*/
if (op == oend)
goto _output_error;
continue;
}
LZ4_SECURECOPY(ref, op, cpy);
op = cpy; /* correction */
}
/* end of decoding */
return (int) (((char *) op) - dest);
/* write overflow error detected */
_output_error:
return (int) (-(((char *) ip) - source));
}
int lz4_decompress(const unsigned char *src, size_t *src_len,
unsigned char *dest, size_t actual_dest_len)
{
int ret = -1;
int input_len = 0;
input_len = lz4_uncompress(src, dest, actual_dest_len);
if (input_len < 0)
goto exit_0;
*src_len = input_len;
return 0;
exit_0:
return ret;
}
#ifndef STATIC
EXPORT_SYMBOL_GPL(lz4_decompress);
#endif
int lz4_decompress_unknownoutputsize(const unsigned char *src, size_t src_len,
unsigned char *dest, size_t *dest_len)
{
int ret = -1;
int out_len = 0;
out_len = lz4_uncompress_unknownoutputsize(src, dest, src_len,
*dest_len);
if (out_len < 0)
goto exit_0;
*dest_len = out_len;
return 0;
exit_0:
return ret;
}
#ifndef STATIC
EXPORT_SYMBOL_GPL(lz4_decompress_unknownoutputsize);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("LZ4 Decompressor");
#endif

156
lib/lz4/lz4defs.h Normal file
View File

@ -0,0 +1,156 @@
/*
* lz4defs.h -- architecture specific defines
*
* Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
/*
* Detects 64 bits mode
*/
#if (defined(__x86_64__) || defined(__x86_64) || defined(__amd64__) \
|| defined(__ppc64__) || defined(__LP64__))
#define LZ4_ARCH64 1
#else
#define LZ4_ARCH64 0
#endif
/*
* Architecture-specific macros
*/
#define BYTE u8
typedef struct _U16_S { u16 v; } U16_S;
typedef struct _U32_S { u32 v; } U32_S;
typedef struct _U64_S { u64 v; } U64_S;
#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) \
|| defined(CONFIG_ARM) && __LINUX_ARM_ARCH__ >= 6 \
&& defined(ARM_EFFICIENT_UNALIGNED_ACCESS)
#define A16(x) (((U16_S *)(x))->v)
#define A32(x) (((U32_S *)(x))->v)
#define A64(x) (((U64_S *)(x))->v)
#define PUT4(s, d) (A32(d) = A32(s))
#define PUT8(s, d) (A64(d) = A64(s))
#define LZ4_WRITE_LITTLEENDIAN_16(p, v) \
do { \
A16(p) = v; \
p += 2; \
} while (0)
#else /* CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS */
#define A64(x) get_unaligned((u64 *)&(((U16_S *)(x))->v))
#define A32(x) get_unaligned((u32 *)&(((U16_S *)(x))->v))
#define A16(x) get_unaligned((u16 *)&(((U16_S *)(x))->v))
#define PUT4(s, d) \
put_unaligned(get_unaligned((const u32 *) s), (u32 *) d)
#define PUT8(s, d) \
put_unaligned(get_unaligned((const u64 *) s), (u64 *) d)
#define LZ4_WRITE_LITTLEENDIAN_16(p, v) \
do { \
put_unaligned(v, (u16 *)(p)); \
p += 2; \
} while (0)
#endif
#define COPYLENGTH 8
#define ML_BITS 4
#define ML_MASK ((1U << ML_BITS) - 1)
#define RUN_BITS (8 - ML_BITS)
#define RUN_MASK ((1U << RUN_BITS) - 1)
#define MEMORY_USAGE 14
#define MINMATCH 4
#define SKIPSTRENGTH 6
#define LASTLITERALS 5
#define MFLIMIT (COPYLENGTH + MINMATCH)
#define MINLENGTH (MFLIMIT + 1)
#define MAXD_LOG 16
#define MAXD (1 << MAXD_LOG)
#define MAXD_MASK (u32)(MAXD - 1)
#define MAX_DISTANCE (MAXD - 1)
#define HASH_LOG (MAXD_LOG - 1)
#define HASHTABLESIZE (1 << HASH_LOG)
#define MAX_NB_ATTEMPTS 256
#define OPTIMAL_ML (int)((ML_MASK-1)+MINMATCH)
#define LZ4_64KLIMIT ((1<<16) + (MFLIMIT - 1))
#define HASHLOG64K ((MEMORY_USAGE - 2) + 1)
#define HASH64KTABLESIZE (1U << HASHLOG64K)
#define LZ4_HASH_VALUE(p) (((A32(p)) * 2654435761U) >> \
((MINMATCH * 8) - (MEMORY_USAGE-2)))
#define LZ4_HASH64K_VALUE(p) (((A32(p)) * 2654435761U) >> \
((MINMATCH * 8) - HASHLOG64K))
#define HASH_VALUE(p) (((A32(p)) * 2654435761U) >> \
((MINMATCH * 8) - HASH_LOG))
#if LZ4_ARCH64/* 64-bit */
#define STEPSIZE 8
#define LZ4_COPYSTEP(s, d) \
do { \
PUT8(s, d); \
d += 8; \
s += 8; \
} while (0)
#define LZ4_COPYPACKET(s, d) LZ4_COPYSTEP(s, d)
#define LZ4_SECURECOPY(s, d, e) \
do { \
if (d < e) { \
LZ4_WILDCOPY(s, d, e); \
} \
} while (0)
#define HTYPE u32
#ifdef __BIG_ENDIAN
#define LZ4_NBCOMMONBYTES(val) (__builtin_clzll(val) >> 3)
#else
#define LZ4_NBCOMMONBYTES(val) (__builtin_ctzll(val) >> 3)
#endif
#else /* 32-bit */
#define STEPSIZE 4
#define LZ4_COPYSTEP(s, d) \
do { \
PUT4(s, d); \
d += 4; \
s += 4; \
} while (0)
#define LZ4_COPYPACKET(s, d) \
do { \
LZ4_COPYSTEP(s, d); \
LZ4_COPYSTEP(s, d); \
} while (0)
#define LZ4_SECURECOPY LZ4_WILDCOPY
#define HTYPE const u8*
#ifdef __BIG_ENDIAN
#define LZ4_NBCOMMONBYTES(val) (__builtin_clz(val) >> 3)
#else
#define LZ4_NBCOMMONBYTES(val) (__builtin_ctz(val) >> 3)
#endif
#endif
#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
(d = s - get_unaligned_le16(p))
#define LZ4_WILDCOPY(s, d, e) \
do { \
LZ4_COPYPACKET(s, d); \
} while (d < e)
#define LZ4_BLINDCOPY(s, d, l) \
do { \
u8 *e = (d) + l; \
LZ4_WILDCOPY(s, d, e); \
d = e; \
} while (0)

539
lib/lz4/lz4hc_compress.c Normal file
View File

@ -0,0 +1,539 @@
/*
* LZ4 HC - High Compression Mode of LZ4
* Copyright (C) 2011-2012, Yann Collet.
* BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* You can contact the author at :
* - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
* - LZ4 source repository : http://code.google.com/p/lz4/
*
* Changed for kernel use by:
* Chanho Min <chanho.min@lge.com>
*/
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/lz4.h>
#include <asm/unaligned.h>
#include "lz4defs.h"
struct lz4hc_data {
const u8 *base;
HTYPE hashtable[HASHTABLESIZE];
u16 chaintable[MAXD];
const u8 *nexttoupdate;
} __attribute__((__packed__));
static inline int lz4hc_init(struct lz4hc_data *hc4, const u8 *base)
{
memset((void *)hc4->hashtable, 0, sizeof(hc4->hashtable));
memset(hc4->chaintable, 0xFF, sizeof(hc4->chaintable));
#if LZ4_ARCH64
hc4->nexttoupdate = base + 1;
#else
hc4->nexttoupdate = base;
#endif
hc4->base = base;
return 1;
}
/* Update chains up to ip (excluded) */
static inline void lz4hc_insert(struct lz4hc_data *hc4, const u8 *ip)
{
u16 *chaintable = hc4->chaintable;
HTYPE *hashtable = hc4->hashtable;
#if LZ4_ARCH64
const BYTE * const base = hc4->base;
#else
const int base = 0;
#endif
while (hc4->nexttoupdate < ip) {
const u8 *p = hc4->nexttoupdate;
size_t delta = p - (hashtable[HASH_VALUE(p)] + base);
if (delta > MAX_DISTANCE)
delta = MAX_DISTANCE;
chaintable[(size_t)(p) & MAXD_MASK] = (u16)delta;
hashtable[HASH_VALUE(p)] = (p) - base;
hc4->nexttoupdate++;
}
}
static inline size_t lz4hc_commonlength(const u8 *p1, const u8 *p2,
const u8 *const matchlimit)
{
const u8 *p1t = p1;
while (p1t < matchlimit - (STEPSIZE - 1)) {
#if LZ4_ARCH64
u64 diff = A64(p2) ^ A64(p1t);
#else
u32 diff = A32(p2) ^ A32(p1t);
#endif
if (!diff) {
p1t += STEPSIZE;
p2 += STEPSIZE;
continue;
}
p1t += LZ4_NBCOMMONBYTES(diff);
return p1t - p1;
}
#if LZ4_ARCH64
if ((p1t < (matchlimit-3)) && (A32(p2) == A32(p1t))) {
p1t += 4;
p2 += 4;
}
#endif
if ((p1t < (matchlimit - 1)) && (A16(p2) == A16(p1t))) {
p1t += 2;
p2 += 2;
}
if ((p1t < matchlimit) && (*p2 == *p1t))
p1t++;
return p1t - p1;
}
static inline int lz4hc_insertandfindbestmatch(struct lz4hc_data *hc4,
const u8 *ip, const u8 *const matchlimit, const u8 **matchpos)
{
u16 *const chaintable = hc4->chaintable;
HTYPE *const hashtable = hc4->hashtable;
const u8 *ref;
#if LZ4_ARCH64
const BYTE * const base = hc4->base;
#else
const int base = 0;
#endif
int nbattempts = MAX_NB_ATTEMPTS;
size_t repl = 0, ml = 0;
u16 delta;
/* HC4 match finder */
lz4hc_insert(hc4, ip);
ref = hashtable[HASH_VALUE(ip)] + base;
/* potential repetition */
if (ref >= ip-4) {
/* confirmed */
if (A32(ref) == A32(ip)) {
delta = (u16)(ip-ref);
repl = ml = lz4hc_commonlength(ip + MINMATCH,
ref + MINMATCH, matchlimit) + MINMATCH;
*matchpos = ref;
}
ref -= (size_t)chaintable[(size_t)(ref) & MAXD_MASK];
}
while ((ref >= ip - MAX_DISTANCE) && nbattempts) {
nbattempts--;
if (*(ref + ml) == *(ip + ml)) {
if (A32(ref) == A32(ip)) {
size_t mlt =
lz4hc_commonlength(ip + MINMATCH,
ref + MINMATCH, matchlimit) + MINMATCH;
if (mlt > ml) {
ml = mlt;
*matchpos = ref;
}
}
}
ref -= (size_t)chaintable[(size_t)(ref) & MAXD_MASK];
}
/* Complete table */
if (repl) {
const BYTE *ptr = ip;
const BYTE *end;
end = ip + repl - (MINMATCH-1);
/* Pre-Load */
while (ptr < end - delta) {
chaintable[(size_t)(ptr) & MAXD_MASK] = delta;
ptr++;
}
do {
chaintable[(size_t)(ptr) & MAXD_MASK] = delta;
/* Head of chain */
hashtable[HASH_VALUE(ptr)] = (ptr) - base;
ptr++;
} while (ptr < end);
hc4->nexttoupdate = end;
}
return (int)ml;
}
static inline int lz4hc_insertandgetwidermatch(struct lz4hc_data *hc4,
const u8 *ip, const u8 *startlimit, const u8 *matchlimit, int longest,
const u8 **matchpos, const u8 **startpos)
{
u16 *const chaintable = hc4->chaintable;
HTYPE *const hashtable = hc4->hashtable;
#if LZ4_ARCH64
const BYTE * const base = hc4->base;
#else
const int base = 0;
#endif
const u8 *ref;
int nbattempts = MAX_NB_ATTEMPTS;
int delta = (int)(ip - startlimit);
/* First Match */
lz4hc_insert(hc4, ip);
ref = hashtable[HASH_VALUE(ip)] + base;
while ((ref >= ip - MAX_DISTANCE) && (ref >= hc4->base)
&& (nbattempts)) {
nbattempts--;
if (*(startlimit + longest) == *(ref - delta + longest)) {
if (A32(ref) == A32(ip)) {
const u8 *reft = ref + MINMATCH;
const u8 *ipt = ip + MINMATCH;
const u8 *startt = ip;
while (ipt < matchlimit-(STEPSIZE - 1)) {
#if LZ4_ARCH64
u64 diff = A64(reft) ^ A64(ipt);
#else
u32 diff = A32(reft) ^ A32(ipt);
#endif
if (!diff) {
ipt += STEPSIZE;
reft += STEPSIZE;
continue;
}
ipt += LZ4_NBCOMMONBYTES(diff);
goto _endcount;
}
#if LZ4_ARCH64
if ((ipt < (matchlimit - 3))
&& (A32(reft) == A32(ipt))) {
ipt += 4;
reft += 4;
}
ipt += 2;
#endif
if ((ipt < (matchlimit - 1))
&& (A16(reft) == A16(ipt))) {
reft += 2;
}
if ((ipt < matchlimit) && (*reft == *ipt))
ipt++;
_endcount:
reft = ref;
while ((startt > startlimit)
&& (reft > hc4->base)
&& (startt[-1] == reft[-1])) {
startt--;
reft--;
}
if ((ipt - startt) > longest) {
longest = (int)(ipt - startt);
*matchpos = reft;
*startpos = startt;
}
}
}
ref -= (size_t)chaintable[(size_t)(ref) & MAXD_MASK];
}
return longest;
}
static inline int lz4_encodesequence(const u8 **ip, u8 **op, const u8 **anchor,
int ml, const u8 *ref)
{
int length, len;
u8 *token;
/* Encode Literal length */
length = (int)(*ip - *anchor);
token = (*op)++;
if (length >= (int)RUN_MASK) {
*token = (RUN_MASK << ML_BITS);
len = length - RUN_MASK;
for (; len > 254 ; len -= 255)
*(*op)++ = 255;
*(*op)++ = (u8)len;
} else
*token = (length << ML_BITS);
/* Copy Literals */
LZ4_BLINDCOPY(*anchor, *op, length);
/* Encode Offset */
LZ4_WRITE_LITTLEENDIAN_16(*op, (u16)(*ip - ref));
/* Encode MatchLength */
len = (int)(ml - MINMATCH);
if (len >= (int)ML_MASK) {
*token += ML_MASK;
len -= ML_MASK;
for (; len > 509 ; len -= 510) {
*(*op)++ = 255;
*(*op)++ = 255;
}
if (len > 254) {
len -= 255;
*(*op)++ = 255;
}
*(*op)++ = (u8)len;
} else
*token += len;
/* Prepare next loop */
*ip += ml;
*anchor = *ip;
return 0;
}
static int lz4_compresshcctx(struct lz4hc_data *ctx,
const char *source,
char *dest,
int isize)
{
const u8 *ip = (const u8 *)source;
const u8 *anchor = ip;
const u8 *const iend = ip + isize;
const u8 *const mflimit = iend - MFLIMIT;
const u8 *const matchlimit = (iend - LASTLITERALS);
u8 *op = (u8 *)dest;
int ml, ml2, ml3, ml0;
const u8 *ref = NULL;
const u8 *start2 = NULL;
const u8 *ref2 = NULL;
const u8 *start3 = NULL;
const u8 *ref3 = NULL;
const u8 *start0;
const u8 *ref0;
int lastrun;
ip++;
/* Main Loop */
while (ip < mflimit) {
ml = lz4hc_insertandfindbestmatch(ctx, ip, matchlimit, (&ref));
if (!ml) {
ip++;
continue;
}
/* saved, in case we would skip too much */
start0 = ip;
ref0 = ref;
ml0 = ml;
_search2:
if (ip+ml < mflimit)
ml2 = lz4hc_insertandgetwidermatch(ctx, ip + ml - 2,
ip + 1, matchlimit, ml, &ref2, &start2);
else
ml2 = ml;
/* No better match */
if (ml2 == ml) {
lz4_encodesequence(&ip, &op, &anchor, ml, ref);
continue;
}
if (start0 < ip) {
/* empirical */
if (start2 < ip + ml0) {
ip = start0;
ref = ref0;
ml = ml0;
}
}
/*
* Here, start0==ip
* First Match too small : removed
*/
if ((start2 - ip) < 3) {
ml = ml2;
ip = start2;
ref = ref2;
goto _search2;
}
_search3:
/*
* Currently we have :
* ml2 > ml1, and
* ip1+3 <= ip2 (usually < ip1+ml1)
*/
if ((start2 - ip) < OPTIMAL_ML) {
int correction;
int new_ml = ml;
if (new_ml > OPTIMAL_ML)
new_ml = OPTIMAL_ML;
if (ip + new_ml > start2 + ml2 - MINMATCH)
new_ml = (int)(start2 - ip) + ml2 - MINMATCH;
correction = new_ml - (int)(start2 - ip);
if (correction > 0) {
start2 += correction;
ref2 += correction;
ml2 -= correction;
}
}
/*
* Now, we have start2 = ip+new_ml,
* with new_ml=min(ml, OPTIMAL_ML=18)
*/
if (start2 + ml2 < mflimit)
ml3 = lz4hc_insertandgetwidermatch(ctx,
start2 + ml2 - 3, start2, matchlimit,
ml2, &ref3, &start3);
else
ml3 = ml2;
/* No better match : 2 sequences to encode */
if (ml3 == ml2) {
/* ip & ref are known; Now for ml */
if (start2 < ip+ml)
ml = (int)(start2 - ip);
/* Now, encode 2 sequences */
lz4_encodesequence(&ip, &op, &anchor, ml, ref);
ip = start2;
lz4_encodesequence(&ip, &op, &anchor, ml2, ref2);
continue;
}
/* Not enough space for match 2 : remove it */
if (start3 < ip + ml + 3) {
/*
* can write Seq1 immediately ==> Seq2 is removed,
* so Seq3 becomes Seq1
*/
if (start3 >= (ip + ml)) {
if (start2 < ip + ml) {
int correction =
(int)(ip + ml - start2);
start2 += correction;
ref2 += correction;
ml2 -= correction;
if (ml2 < MINMATCH) {
start2 = start3;
ref2 = ref3;
ml2 = ml3;
}
}
lz4_encodesequence(&ip, &op, &anchor, ml, ref);
ip = start3;
ref = ref3;
ml = ml3;
start0 = start2;
ref0 = ref2;
ml0 = ml2;
goto _search2;
}
start2 = start3;
ref2 = ref3;
ml2 = ml3;
goto _search3;
}
/*
* OK, now we have 3 ascending matches; let's write at least
* the first one ip & ref are known; Now for ml
*/
if (start2 < ip + ml) {
if ((start2 - ip) < (int)ML_MASK) {
int correction;
if (ml > OPTIMAL_ML)
ml = OPTIMAL_ML;
if (ip + ml > start2 + ml2 - MINMATCH)
ml = (int)(start2 - ip) + ml2
- MINMATCH;
correction = ml - (int)(start2 - ip);
if (correction > 0) {
start2 += correction;
ref2 += correction;
ml2 -= correction;
}
} else
ml = (int)(start2 - ip);
}
lz4_encodesequence(&ip, &op, &anchor, ml, ref);
ip = start2;
ref = ref2;
ml = ml2;
start2 = start3;
ref2 = ref3;
ml2 = ml3;
goto _search3;
}
/* Encode Last Literals */
lastrun = (int)(iend - anchor);
if (lastrun >= (int)RUN_MASK) {
*op++ = (RUN_MASK << ML_BITS);
lastrun -= RUN_MASK;
for (; lastrun > 254 ; lastrun -= 255)
*op++ = 255;
*op++ = (u8) lastrun;
} else
*op++ = (lastrun << ML_BITS);
memcpy(op, anchor, iend - anchor);
op += iend - anchor;
/* End */
return (int) (((char *)op) - dest);
}
int lz4hc_compress(const unsigned char *src, size_t src_len,
unsigned char *dst, size_t *dst_len, void *wrkmem)
{
int ret = -1;
int out_len = 0;
struct lz4hc_data *hc4 = (struct lz4hc_data *)wrkmem;
lz4hc_init(hc4, (const u8 *)src);
out_len = lz4_compresshcctx((struct lz4hc_data *)hc4, (const u8 *)src,
(char *)dst, (int)src_len);
if (out_len < 0)
goto exit;
*dst_len = out_len;
return 0;
exit:
return ret;
}
EXPORT_SYMBOL_GPL(lz4hc_compress);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("LZ4HC compressor");

32
mm/Kconfig
View File

@ -504,3 +504,35 @@ config CMA_DEBUG
messages for every CMA call as well as various messages while
processing calls such as dma_alloc_from_contiguous().
This option does not affect warning and error messages.
config ZPOOL
tristate "Common API for compressed memory storage"
default n
help
Compressed memory storage API. This allows using either zbud or
zsmalloc.
config ZSMALLOC
bool "Memory allocator for compressed pages"
depends on MMU
default n
help
zsmalloc is a slab-based memory allocator designed to store
compressed RAM pages. zsmalloc uses virtual memory mapping
in order to reduce fragmentation. However, this results in a
non-standard allocator interface where a handle, not a pointer, is
returned by an alloc(). This handle must be mapped in order to
access the allocated space.
config PGTABLE_MAPPING
bool "Use page table mapping to access object in zsmalloc"
depends on ZSMALLOC
help
By default, zsmalloc uses a copy-based object mapping method to
access allocations that span two pages. However, if a particular
architecture (ex, ARM) performs VM mapping faster than copying,
then you should select this. This causes zsmalloc to use page table
mapping rather than copying for object mapping.
You can check speed with zsmalloc benchmark[1].
[1] https://github.com/spartacus06/zsmalloc

2
mm/Makefile
View File

@ -59,3 +59,5 @@ obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o
obj-$(CONFIG_CLEANCACHE) += cleancache.o
obj-$(CONFIG_MEMORY_ISOLATION) += page_isolation.o
obj-$(CONFIG_GENERIC_EARLY_IOREMAP) += early_ioremap.o
obj-$(CONFIG_ZPOOL) += zpool.o
obj-$(CONFIG_ZSMALLOC) += zsmalloc.o

364
mm/zpool.c Normal file
View File

@ -0,0 +1,364 @@
/*
* zpool memory storage api
*
* Copyright (C) 2014 Dan Streetman
*
* This is a common frontend for memory storage pool implementations.
* Typically, this is used to store compressed memory.
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/list.h>
#include <linux/types.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/module.h>
#include <linux/zpool.h>
struct zpool {
char *type;
struct zpool_driver *driver;
void *pool;
struct zpool_ops *ops;
struct list_head list;
};
static LIST_HEAD(drivers_head);
static DEFINE_SPINLOCK(drivers_lock);
static LIST_HEAD(pools_head);
static DEFINE_SPINLOCK(pools_lock);
/**
* zpool_register_driver() - register a zpool implementation.
* @driver: driver to register
*/
void zpool_register_driver(struct zpool_driver *driver)
{
spin_lock(&drivers_lock);
atomic_set(&driver->refcount, 0);
list_add(&driver->list, &drivers_head);
spin_unlock(&drivers_lock);
}
EXPORT_SYMBOL(zpool_register_driver);
/**
* zpool_unregister_driver() - unregister a zpool implementation.
* @driver: driver to unregister.
*
* Module usage counting is used to prevent using a driver
* while/after unloading, so if this is called from module
* exit function, this should never fail; if called from
* other than the module exit function, and this returns
* failure, the driver is in use and must remain available.
*/
int zpool_unregister_driver(struct zpool_driver *driver)
{
int ret = 0, refcount;
spin_lock(&drivers_lock);
refcount = atomic_read(&driver->refcount);
WARN_ON(refcount < 0);
if (refcount > 0)
ret = -EBUSY;
else
list_del(&driver->list);
spin_unlock(&drivers_lock);
return ret;
}
EXPORT_SYMBOL(zpool_unregister_driver);
/**
* zpool_evict() - evict callback from a zpool implementation.
* @pool: pool to evict from.
* @handle: handle to evict.
*
* This can be used by zpool implementations to call the
* user's evict zpool_ops struct evict callback.
*/
int zpool_evict(void *pool, unsigned long handle)
{
struct zpool *zpool;
spin_lock(&pools_lock);
list_for_each_entry(zpool, &pools_head, list) {
if (zpool->pool == pool) {
spin_unlock(&pools_lock);
if (!zpool->ops || !zpool->ops->evict)
return -EINVAL;
return zpool->ops->evict(zpool, handle);
}
}
spin_unlock(&pools_lock);
return -ENOENT;
}
EXPORT_SYMBOL(zpool_evict);
static struct zpool_driver *zpool_get_driver(char *type)
{
struct zpool_driver *driver;
spin_lock(&drivers_lock);
list_for_each_entry(driver, &drivers_head, list) {
if (!strcmp(driver->type, type)) {
bool got = try_module_get(driver->owner);
if (got)
atomic_inc(&driver->refcount);
spin_unlock(&drivers_lock);
return got ? driver : NULL;
}
}
spin_unlock(&drivers_lock);
return NULL;
}
static void zpool_put_driver(struct zpool_driver *driver)
{
atomic_dec(&driver->refcount);
module_put(driver->owner);
}
/**
* zpool_create_pool() - Create a new zpool
* @type The type of the zpool to create (e.g. zbud, zsmalloc)
* @gfp The GFP flags to use when allocating the pool.
* @ops The optional ops callback.
*
* This creates a new zpool of the specified type. The gfp flags will be
* used when allocating memory, if the implementation supports it. If the
* ops param is NULL, then the created zpool will not be shrinkable.
*
* Implementations must guarantee this to be thread-safe.
*
* Returns: New zpool on success, NULL on failure.
*/
struct zpool *zpool_create_pool(char *type, gfp_t gfp, struct zpool_ops *ops)
{
struct zpool_driver *driver;
struct zpool *zpool;
pr_info("creating pool type %s\n", type);
driver = zpool_get_driver(type);
if (!driver) {
request_module("zpool-%s", type);
driver = zpool_get_driver(type);
}
if (!driver) {
pr_err("no driver for type %s\n", type);
return NULL;
}
zpool = kmalloc(sizeof(*zpool), gfp);
if (!zpool) {
pr_err("couldn't create zpool - out of memory\n");
zpool_put_driver(driver);
return NULL;
}
zpool->type = driver->type;
zpool->driver = driver;
zpool->pool = driver->create(gfp, ops);
zpool->ops = ops;
if (!zpool->pool) {
pr_err("couldn't create %s pool\n", type);
zpool_put_driver(driver);
kfree(zpool);
return NULL;
}
pr_info("created %s pool\n", type);
spin_lock(&pools_lock);
list_add(&zpool->list, &pools_head);
spin_unlock(&pools_lock);
return zpool;
}
/**
* zpool_destroy_pool() - Destroy a zpool
* @pool The zpool to destroy.
*
* Implementations must guarantee this to be thread-safe,
* however only when destroying different pools. The same
* pool should only be destroyed once, and should not be used
* after it is destroyed.
*
* This destroys an existing zpool. The zpool should not be in use.
*/
void zpool_destroy_pool(struct zpool *zpool)
{
pr_info("destroying pool type %s\n", zpool->type);
spin_lock(&pools_lock);
list_del(&zpool->list);
spin_unlock(&pools_lock);
zpool->driver->destroy(zpool->pool);
zpool_put_driver(zpool->driver);
kfree(zpool);
}
/**
* zpool_get_type() - Get the type of the zpool
* @pool The zpool to check
*
* This returns the type of the pool.
*
* Implementations must guarantee this to be thread-safe.
*
* Returns: The type of zpool.
*/
char *zpool_get_type(struct zpool *zpool)
{
return zpool->type;
}
/**
* zpool_malloc() - Allocate memory
* @pool The zpool to allocate from.
* @size The amount of memory to allocate.
* @gfp The GFP flags to use when allocating memory.
* @handle Pointer to the handle to set
*
* This allocates the requested amount of memory from the pool.
* The gfp flags will be used when allocating memory, if the
* implementation supports it. The provided @handle will be
* set to the allocated object handle.
*
* Implementations must guarantee this to be thread-safe.
*
* Returns: 0 on success, negative value on error.
*/
int zpool_malloc(struct zpool *zpool, size_t size, gfp_t gfp,
unsigned long *handle)
{
return zpool->driver->malloc(zpool->pool, size, gfp, handle);
}
/**
* zpool_free() - Free previously allocated memory
* @pool The zpool that allocated the memory.
* @handle The handle to the memory to free.
*
* This frees previously allocated memory. This does not guarantee
* that the pool will actually free memory, only that the memory
* in the pool will become available for use by the pool.
*
* Implementations must guarantee this to be thread-safe,
* however only when freeing different handles. The same
* handle should only be freed once, and should not be used
* after freeing.
*/
void zpool_free(struct zpool *zpool, unsigned long handle)
{
zpool->driver->free(zpool->pool, handle);
}
/**
* zpool_shrink() - Shrink the pool size
* @pool The zpool to shrink.
* @pages The number of pages to shrink the pool.
* @reclaimed The number of pages successfully evicted.
*
* This attempts to shrink the actual memory size of the pool
* by evicting currently used handle(s). If the pool was
* created with no zpool_ops, or the evict call fails for any
* of the handles, this will fail. If non-NULL, the @reclaimed
* parameter will be set to the number of pages reclaimed,
* which may be more than the number of pages requested.
*
* Implementations must guarantee this to be thread-safe.
*
* Returns: 0 on success, negative value on error/failure.
*/
int zpool_shrink(struct zpool *zpool, unsigned int pages,
unsigned int *reclaimed)
{
return zpool->driver->shrink(zpool->pool, pages, reclaimed);
}
/**
* zpool_map_handle() - Map a previously allocated handle into memory
* @pool The zpool that the handle was allocated from
* @handle The handle to map
* @mm How the memory should be mapped
*
* This maps a previously allocated handle into memory. The @mm
* param indicates to the implementation how the memory will be
* used, i.e. read-only, write-only, read-write. If the
* implementation does not support it, the memory will be treated
* as read-write.
*
* This may hold locks, disable interrupts, and/or preemption,
* and the zpool_unmap_handle() must be called to undo those
* actions. The code that uses the mapped handle should complete
* its operatons on the mapped handle memory quickly and unmap
* as soon as possible. As the implementation may use per-cpu
* data, multiple handles should not be mapped concurrently on
* any cpu.
*
* Returns: A pointer to the handle's mapped memory area.
*/
void *zpool_map_handle(struct zpool *zpool, unsigned long handle,
enum zpool_mapmode mapmode)
{
return zpool->driver->map(zpool->pool, handle, mapmode);
}
/**
* zpool_unmap_handle() - Unmap a previously mapped handle
* @pool The zpool that the handle was allocated from
* @handle The handle to unmap
*
* This unmaps a previously mapped handle. Any locks or other
* actions that the implementation took in zpool_map_handle()
* will be undone here. The memory area returned from
* zpool_map_handle() should no longer be used after this.
*/
void zpool_unmap_handle(struct zpool *zpool, unsigned long handle)
{
zpool->driver->unmap(zpool->pool, handle);
}
/**
* zpool_get_total_size() - The total size of the pool
* @pool The zpool to check
*
* This returns the total size in bytes of the pool.
*
* Returns: Total size of the zpool in bytes.
*/
u64 zpool_get_total_size(struct zpool *zpool)
{
return zpool->driver->total_size(zpool->pool);
}
static int __init init_zpool(void)
{
pr_info("loaded\n");
return 0;
}
static void __exit exit_zpool(void)
{
pr_info("unloaded\n");
}
module_init(init_zpool);
module_exit(exit_zpool);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Dan Streetman <ddstreet@ieee.org>");
MODULE_DESCRIPTION("Common API for compressed memory storage");

274
drivers/staging/zsmalloc/zsmalloc-main.c → mm/zsmalloc.c
View File

@ -2,6 +2,7 @@
* zsmalloc memory allocator
*
* Copyright (C) 2011 Nitin Gupta
* Copyright (C) 2012, 2013 Minchan Kim
*
* This code is released using a dual license strategy: BSD/GPL
* You can choose the license that better fits your requirements.
@ -78,8 +79,8 @@
#include <linux/hardirq.h>
#include <linux/spinlock.h>
#include <linux/types.h>
#include "zsmalloc.h"
#include <linux/zsmalloc.h>
#include <linux/zpool.h>
/*
* This must be power of 2 and greater than of equal to sizeof(link_free).
@ -129,7 +130,7 @@
#define ZS_MAX_ALLOC_SIZE PAGE_SIZE
/*
* On systems with 4K page size, this gives 254 size classes! There is a
* On systems with 4K page size, this gives 255 size classes! There is a
* trader-off here:
* - Large number of size classes is potentially wasteful as free page are
* spread across these classes
@ -162,7 +163,7 @@ enum fullness_group {
* n <= N / f, where
* n = number of allocated objects
* N = total number of objects zspage can store
* f = 1/fullness_threshold_frac
* f = fullness_threshold_frac
*
* Similarly, we assign zspage to:
* ZS_ALMOST_FULL when n > N / f
@ -186,9 +187,6 @@ struct size_class {
spinlock_t lock;
/* stats */
u64 pages_allocated;
struct page *fullness_list[_ZS_NR_FULLNESS_GROUPS];
};
@ -204,9 +202,10 @@ struct link_free {
};
struct zs_pool {
struct size_class size_class[ZS_SIZE_CLASSES];
struct size_class *size_class[ZS_SIZE_CLASSES];
gfp_t flags; /* allocation flags used when growing pool */
atomic_long_t pages_allocated;
};
/*
@ -239,6 +238,82 @@ struct mapping_area {
enum zs_mapmode vm_mm; /* mapping mode */
};
/* zpool driver */
#ifdef CONFIG_ZPOOL
static void *zs_zpool_create(gfp_t gfp, struct zpool_ops *zpool_ops)
{
return zs_create_pool(gfp);
}
static void zs_zpool_destroy(void *pool)
{
zs_destroy_pool(pool);
}
static int zs_zpool_malloc(void *pool, size_t size, gfp_t gfp,
unsigned long *handle)
{
*handle = zs_malloc(pool, size);
return *handle ? 0 : -1;
}
static void zs_zpool_free(void *pool, unsigned long handle)
{
zs_free(pool, handle);
}
static int zs_zpool_shrink(void *pool, unsigned int pages,
unsigned int *reclaimed)
{
return -EINVAL;
}
static void *zs_zpool_map(void *pool, unsigned long handle,
enum zpool_mapmode mm)
{
enum zs_mapmode zs_mm;
switch (mm) {
case ZPOOL_MM_RO:
zs_mm = ZS_MM_RO;
break;
case ZPOOL_MM_WO:
zs_mm = ZS_MM_WO;
break;
case ZPOOL_MM_RW: /* fallthru */
default:
zs_mm = ZS_MM_RW;
break;
}
return zs_map_object(pool, handle, zs_mm);
}
static void zs_zpool_unmap(void *pool, unsigned long handle)
{
zs_unmap_object(pool, handle);
}
static u64 zs_zpool_total_size(void *pool)
{
return zs_get_total_pages(pool) << PAGE_SHIFT;
}
static struct zpool_driver zs_zpool_driver = {
.type = "zsmalloc",
.owner = THIS_MODULE,
.create = zs_zpool_create,
.destroy = zs_zpool_destroy,
.malloc = zs_zpool_malloc,
.free = zs_zpool_free,
.shrink = zs_zpool_shrink,
.map = zs_zpool_map,
.unmap = zs_zpool_unmap,
.total_size = zs_zpool_total_size,
};
MODULE_ALIAS("zpool-zsmalloc");
#endif /* CONFIG_ZPOOL */
/* per-cpu VM mapping areas for zspage accesses that cross page boundaries */
static DEFINE_PER_CPU(struct mapping_area, zs_map_area);
@ -359,7 +434,7 @@ static enum fullness_group fix_fullness_group(struct zs_pool *pool,
if (newfg == currfg)
goto out;
class = &pool->size_class[class_idx];
class = pool->size_class[class_idx];
remove_zspage(page, class, currfg);
insert_zspage(page, class, newfg);
set_zspage_mapping(page, class_idx, newfg);
@ -519,7 +594,8 @@ static void init_zspage(struct page *first_page, struct size_class *class)
while (page) {
struct page *next_page;
struct link_free *link;
unsigned int i, objs_on_page;
unsigned int i = 1;
void *vaddr;
/*
* page->index stores offset of first object starting
@ -530,16 +606,12 @@ static void init_zspage(struct page *first_page, struct size_class *class)
if (page != first_page)
page->index = off;
link = (struct link_free *)kmap_atomic(page) +
off / sizeof(*link);
objs_on_page = (PAGE_SIZE - off) / class->size;
vaddr = kmap_atomic(page);
link = (struct link_free *)vaddr + off / sizeof(*link);
for (i = 1; i <= objs_on_page; i++) {
off += class->size;
if (off < PAGE_SIZE) {
link->next = obj_location_to_handle(page, i);
link += class->size / sizeof(*link);
}
while ((off += class->size) < PAGE_SIZE) {
link->next = obj_location_to_handle(page, i++);
link += class->size / sizeof(*link);
}
/*
@ -549,9 +621,9 @@ static void init_zspage(struct page *first_page, struct size_class *class)
*/
next_page = get_next_page(page);
link->next = obj_location_to_handle(next_page, 0);
kunmap_atomic(link);
kunmap_atomic(vaddr);
page = next_page;
off = (off + class->size) % PAGE_SIZE;
off %= PAGE_SIZE;
}
}
@ -776,31 +848,76 @@ static struct notifier_block zs_cpu_nb = {
.notifier_call = zs_cpu_notifier
};
static void zs_exit(void)
static void zs_unregister_cpu_notifier(void)
{
int cpu;
cpu_notifier_register_begin();
for_each_online_cpu(cpu)
zs_cpu_notifier(NULL, CPU_DEAD, (void *)(long)cpu);
unregister_cpu_notifier(&zs_cpu_nb);
__unregister_cpu_notifier(&zs_cpu_nb);
cpu_notifier_register_done();
}
static int zs_init(void)
static int zs_register_cpu_notifier(void)
{
int cpu, ret;
int cpu, uninitialized_var(ret);
register_cpu_notifier(&zs_cpu_nb);
cpu_notifier_register_begin();
__register_cpu_notifier(&zs_cpu_nb);
for_each_online_cpu(cpu) {
ret = zs_cpu_notifier(NULL, CPU_UP_PREPARE, (void *)(long)cpu);
if (notifier_to_errno(ret))
goto fail;
break;
}
return 0;
fail:
zs_exit();
cpu_notifier_register_done();
return notifier_to_errno(ret);
}
static void __exit zs_exit(void)
{
#ifdef CONFIG_ZPOOL
zpool_unregister_driver(&zs_zpool_driver);
#endif
zs_unregister_cpu_notifier();
}
static int __init zs_init(void)
{
int ret = zs_register_cpu_notifier();
if (ret) {
zs_unregister_cpu_notifier();
return ret;
}
#ifdef CONFIG_ZPOOL
zpool_register_driver(&zs_zpool_driver);
#endif
return 0;
}
static unsigned int get_maxobj_per_zspage(int size, int pages_per_zspage)
{
return pages_per_zspage * PAGE_SIZE / size;
}
static bool can_merge(struct size_class *prev, int size, int pages_per_zspage)
{
if (prev->pages_per_zspage != pages_per_zspage)
return false;
if (get_maxobj_per_zspage(prev->size, prev->pages_per_zspage)
!= get_maxobj_per_zspage(size, pages_per_zspage))
return false;
return true;
}
/**
* zs_create_pool - Creates an allocation pool to work from.
* @flags: allocation flags used to allocate pool metadata
@ -821,25 +938,56 @@ struct zs_pool *zs_create_pool(gfp_t flags)
if (!pool)
return NULL;
for (i = 0; i < ZS_SIZE_CLASSES; i++) {
/*
* Iterate reversly, because, size of size_class that we want to use
* for merging should be larger or equal to current size.
*/
for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) {
int size;
int pages_per_zspage;
struct size_class *class;
struct size_class *prev_class;
size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA;
if (size > ZS_MAX_ALLOC_SIZE)
size = ZS_MAX_ALLOC_SIZE;
pages_per_zspage = get_pages_per_zspage(size);
/*
* size_class is used for normal zsmalloc operation such
* as alloc/free for that size. Although it is natural that we
* have one size_class for each size, there is a chance that we
* can get more memory utilization if we use one size_class for
* many different sizes whose size_class have same
* characteristics. So, we makes size_class point to
* previous size_class if possible.
*/
if (i < ZS_SIZE_CLASSES - 1) {
prev_class = pool->size_class[i + 1];
if (can_merge(prev_class, size, pages_per_zspage)) {
pool->size_class[i] = prev_class;
continue;
}
}
class = kzalloc(sizeof(struct size_class), GFP_KERNEL);
if (!class)
goto err;
class = &pool->size_class[i];
class->size = size;
class->index = i;
class->pages_per_zspage = pages_per_zspage;
spin_lock_init(&class->lock);
class->pages_per_zspage = get_pages_per_zspage(size);
pool->size_class[i] = class;
}
pool->flags = flags;
return pool;
err:
zs_destroy_pool(pool);
return NULL;
}
EXPORT_SYMBOL_GPL(zs_create_pool);
@ -849,7 +997,13 @@ void zs_destroy_pool(struct zs_pool *pool)
for (i = 0; i < ZS_SIZE_CLASSES; i++) {
int fg;
struct size_class *class = &pool->size_class[i];
struct size_class *class = pool->size_class[i];
if (!class)
continue;
if (class->index != i)
continue;
for (fg = 0; fg < _ZS_NR_FULLNESS_GROUPS; fg++) {
if (class->fullness_list[fg]) {
@ -858,6 +1012,7 @@ void zs_destroy_pool(struct zs_pool *pool)
class->size, fg);
}
}
kfree(class);
}
kfree(pool);
}
@ -876,8 +1031,8 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size)
{
unsigned long obj;
struct link_free *link;
int class_idx;
struct size_class *class;
void *vaddr;
struct page *first_page, *m_page;
unsigned long m_objidx, m_offset;
@ -885,9 +1040,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size)
if (unlikely(!size || size > ZS_MAX_ALLOC_SIZE))
return 0;
class_idx = get_size_class_index(size);
class = &pool->size_class[class_idx];
BUG_ON(class_idx != class->index);
class = pool->size_class[get_size_class_index(size)];
spin_lock(&class->lock);
first_page = find_get_zspage(class);
@ -899,19 +1052,20 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size)
return 0;
set_zspage_mapping(first_page, class->index, ZS_EMPTY);
atomic_long_add(class->pages_per_zspage,
&pool->pages_allocated);
spin_lock(&class->lock);
class->pages_allocated += class->pages_per_zspage;
}
obj = (unsigned long)first_page->freelist;
obj_handle_to_location(obj, &m_page, &m_objidx);
m_offset = obj_idx_to_offset(m_page, m_objidx, class->size);
link = (struct link_free *)kmap_atomic(m_page) +
m_offset / sizeof(*link);
vaddr = kmap_atomic(m_page);
link = (struct link_free *)vaddr + m_offset / sizeof(*link);
first_page->freelist = link->next;
memset(link, POISON_INUSE, sizeof(*link));
kunmap_atomic(link);
kunmap_atomic(vaddr);
first_page->inuse++;
/* Now move the zspage to another fullness group, if required */
@ -927,6 +1081,7 @@ void zs_free(struct zs_pool *pool, unsigned long obj)
struct link_free *link;
struct page *first_page, *f_page;
unsigned long f_objidx, f_offset;
void *vaddr;
int class_idx;
struct size_class *class;
@ -939,28 +1094,27 @@ void zs_free(struct zs_pool *pool, unsigned long obj)
first_page = get_first_page(f_page);
get_zspage_mapping(first_page, &class_idx, &fullness);
class = &pool->size_class[class_idx];
class = pool->size_class[class_idx];
f_offset = obj_idx_to_offset(f_page, f_objidx, class->size);
spin_lock(&class->lock);
/* Insert this object in containing zspage's freelist */
link = (struct link_free *)((unsigned char *)kmap_atomic(f_page)
+ f_offset);
vaddr = kmap_atomic(f_page);
link = (struct link_free *)(vaddr + f_offset);
link->next = first_page->freelist;
kunmap_atomic(link);
kunmap_atomic(vaddr);
first_page->freelist = (void *)obj;
first_page->inuse--;
fullness = fix_fullness_group(pool, first_page);
if (fullness == ZS_EMPTY)
class->pages_allocated -= class->pages_per_zspage;
spin_unlock(&class->lock);
if (fullness == ZS_EMPTY)
if (fullness == ZS_EMPTY) {
atomic_long_sub(class->pages_per_zspage,
&pool->pages_allocated);
free_zspage(first_page);
}
}
EXPORT_SYMBOL_GPL(zs_free);
@ -1001,7 +1155,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
obj_handle_to_location(handle, &page, &obj_idx);
get_zspage_mapping(get_first_page(page), &class_idx, &fg);
class = &pool->size_class[class_idx];
class = pool->size_class[class_idx];
off = obj_idx_to_offset(page, obj_idx, class->size);
area = &get_cpu_var(zs_map_area);
@ -1035,7 +1189,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
obj_handle_to_location(handle, &page, &obj_idx);
get_zspage_mapping(get_first_page(page), &class_idx, &fg);
class = &pool->size_class[class_idx];
class = pool->size_class[class_idx];
off = obj_idx_to_offset(page, obj_idx, class->size);
area = &__get_cpu_var(zs_map_area);
@ -1054,17 +1208,11 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
}
EXPORT_SYMBOL_GPL(zs_unmap_object);
u64 zs_get_total_size_bytes(struct zs_pool *pool)
unsigned long zs_get_total_pages(struct zs_pool *pool)
{
int i;
u64 npages = 0;
for (i = 0; i < ZS_SIZE_CLASSES; i++)
npages += pool->size_class[i].pages_allocated;
return npages << PAGE_SHIFT;
return atomic_long_read(&pool->pages_allocated);
}
EXPORT_SYMBOL_GPL(zs_get_total_size_bytes);
EXPORT_SYMBOL_GPL(zs_get_total_pages);
module_init(zs_init);
module_exit(zs_exit);