github-mirror/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux.git synced 2026-09-14 16:10:02 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity Graph

dma-mapping updates for Linux 7.3:

Browse Source 
- swiotlb: added new configuration option for the default pool size
 (Jagadeesh Pagadala) and reduced overhead for high watermark tracking
 (chenhuguanshen)
 
 - minor code cleanups and improvements (Vova Sharaienko, Honglei Huang
 and Marek Szyprowski)
 -----BEGIN PGP SIGNATURE-----
 
 iHUEABYKAB0WIQSrngzkoBtlA8uaaJ+Jp1EFxbsSRAUCaoxGQgAKCRCJp1EFxbsS
 RFPEAP0eo9usjFcvh0YKTPh6/mXgqxRuTNQZ7i+2lRGEczKcJQEA7mwkgwpiOaKn
 f++mMVOmsPvl2Y7r/5XqBWywwhyygA0=
 =X8pY
 -----END PGP SIGNATURE-----
mergetag object 04a19b35dc
 type commit
 tag dma-mapping-7.3-2026-08-24-2
 tagger Marek Szyprowski <m.szyprowski@samsung.com> 1787582472 +0200
 
 second dma-mapping update for Linux 7.3:
 
 - important dma-mapping update for confidential-computing, which adds
 proper tracking of the shared DMA state through direct, pool and swiotlb
 paths (Aneesh Kumar K.V)
 -----BEGIN PGP SIGNATURE-----
 
 iHUEABYKAB0WIQSrngzkoBtlA8uaaJ+Jp1EFxbsSRAUCaoxYzgAKCRCJp1EFxbsS
 RM9bAP4mJuHHzj2DqsKV7QX19uhyzmsHIg+ecjBNRaOdUAgelQD9FsaG/fwrZnRT
 y89H0QUErqLsdmkDqV0zsXfaGzWY4gk=
 =dKjS
 -----END PGP SIGNATURE-----

Merge tags 'dma-mapping-7.3-2026-08-24' and 'dma-mapping-7.3-2026-08-24-2' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux

Pull dma-mapping updates from Marek Szyprowski:

 - swiotlb:
     - new configuration option for the default pool size
       (Jagadeesh Pagadala)
     - reduce overhead for high watermark tracking (chenhuguanshen)

 - minor code cleanups and improvements (Vova Sharaienko, Honglei Huang
   and Marek Szyprowski)

 - add proper tracking of the shared DMA state through direct, pool and
   swiotlb paths (Aneesh Kumar K.V)

   This is important for confidential-computing

* tag 'dma-mapping-7.3-2026-08-24' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux:
  dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS
  MAINTAINERS: update tree for DMA MAPPING HELPERS
  dma/swiotlb: introduce Kconfig option for compile-time default pool size
  dma-direct: Improve readability of the dma_direct_map_sg() for P2PDMA case
  iommu/dma: simplify dma_iova_destroy() and drop the free_iova helper
  dma-coherent: use KiB in DMA allocation logs
  dma-coherent: fix spacing coding style issue

* tag 'dma-mapping-7.3-2026-08-24-2' of git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux: (23 commits)
  swiotlb: remove unused SWIOTLB_FORCE flag
  dma: swiotlb: handle set_memory_decrypted() failures
  dma: swiotlb: free dynamic pools from process context
  dma-direct: rename ret to cpu_addr in alloc helpers
  dma-direct: select DMA address encoding from __DMA_ATTR_ALLOC_CC_SHARED
  dma-direct: set decrypted flag for remapped DMA allocations
  dma-direct: make dma_direct_map_phys() honor DMA_ATTR_CC_SHARED
  dma-direct: Move dma_direct_map_phys() to dma/direct.c
  dma-direct: pass attrs to dma_capable() for DMA_ATTR_CC_SHARED checks
  dma-mapping: make dma_pgprot() honor __DMA_ATTR_ALLOC_CC_SHARED
  dma: swiotlb: track pool encryption state and honor DMA_ATTR_CC_SHARED
  dma: swiotlb: pass mapping attributes by reference
  dma-pool: track decrypted atomic pools and select them via attrs
  dma-direct: use __DMA_ATTR_ALLOC_CC_SHARED in alloc/free paths
  dma-mapping: Add internal shared allocation attribute
  coco: arm64: s390: powerpc: Mark secure guests with CC_ATTR_GUEST_MEM_ENCRYPT
  dma-direct: swiotlb: handle swiotlb alloc/free outside __dma_direct_alloc_pages
  s390: Expose protected virtualization through cc_platform_has()
  swiotlb: Preserve allocation virtual address for dynamic pools
  dma: free atomic pool pages by physical address
  ...
This commit is contained in:
Linus Torvalds 2026-08-24 11:35:46 -07:00
commit 2f43193b88
25 changed files with 980 additions and 421 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Documentation/admin-guide/kernel-parameters.txt
View File

@ -7519,7 +7519,7 @@ Kernel parameters
Execution Facility on pSeries.
swiotlb= [ARM,PPC,MIPS,X86,S390,EARLY]
Format: { <int> [,<int>] | force | noforce }
Format: { <int> [,<int>] | force | noforce | track_hiwater}
<int> -- Number of I/O TLB slabs
<int> -- Second integer after comma. Number of swiotlb
areas with their own lock. Will be rounded up
@ -7527,6 +7527,8 @@ Kernel parameters
force -- force using of bounce buffers even if they
wouldn't be automatically used by the kernel
noforce -- Never use bounce buffers (for debugging)
track_hiwater -- Track high watermark of swiotlb buffers.
Only available when CONFIG_DEBUG_FS is set.
switches= [HW,M68k,EARLY]

29
Documentation/core-api/dma-attributes.rst
View File

@ -179,3 +179,32 @@ interface when building their uAPIs, when possible.
It must never be used in an in-kernel driver that only works with
kernel memory.
DMA_ATTR_CC_SHARED
------------------
This attribute indicates that a DMA mapping is shared, or decrypted, for
confidential computing guests. For normal system memory, the caller must
already have marked the memory decrypted with set_memory_decrypted(). CPU
PTEs for the mapping must use pgprot_decrypted(), and the same shared
semantic may be passed to a vIOMMU when it sets up the IOPTE.
This attribute describes an existing mapping. It does not allocate shared
backing pages and must not be passed to dma_alloc_attrs(). For MMIO, use
this together with DMA_ATTR_MMIO to indicate shared MMIO. Unless
DMA_ATTR_MMIO is provided, the mapping requires a struct page.
__DMA_ATTR_ALLOC_CC_SHARED
--------------------------
This is an internal DMA-mapping attribute for confidential computing guests.
It is used by allocation paths after the DMA core has determined that the
backing pages must be shared, or decrypted. For example, the direct DMA and
SWIOTLB allocation paths use it to select shared DMA pools, decrypt newly
allocated pages, derive DMA addresses using the shared-memory translation, and
restore encryption on free.
__DMA_ATTR_ALLOC_CC_SHARED differs from DMA_ATTR_CC_SHARED in that it is not
a caller-visible DMA API attribute. DMA_ATTR_CC_SHARED describes an
already-shared mapping and requires the caller to have prepared normal
system memory before mapping it.

7
Documentation/core-api/swiotlb.rst
View File

@ -140,8 +140,11 @@ Data structures concepts
------------------------
Memory used for swiotlb bounce buffers is allocated from overall system memory
as one or more "pools". The default pool is allocated during system boot with a
default size of 64 MiB. The default pool size may be modified with the
"swiotlb=" kernel boot line parameter. The default size may also be adjusted
default size of 64 MiB, which can be changed at compile time via
CONFIG_SWIOTLB_DEFAULT_SIZE_MB. The default pool size may also be
modified at runtime with the "swiotlb=" kernel boot line parameter,
which takes precedence over the compile-time default. The default size
may also be adjusted
due to other conditions, such as running in a CoCo VM, as described above. If
CONFIG_SWIOTLB_DYNAMIC is enabled, additional pools may be allocated later in
the life of the system. Each pool must be a contiguous range of physical

3
MAINTAINERS
View File

@ -7705,8 +7705,7 @@ M: Marek Szyprowski <m.szyprowski@samsung.com>
R: Robin Murphy <robin.murphy@arm.com>
L: iommu@lists.linux.dev
S: Supported
W: http://git.infradead.org/users/hch/dma-mapping.git
T: git git://git.infradead.org/users/hch/dma-mapping.git
T: git git://git.kernel.org/pub/scm/linux/kernel/git/mszyprowski/linux.git
F: include/asm-generic/dma-mapping.h
F: include/linux/dma-direct.h
F: include/linux/dma-map-ops.h

5
arch/arm64/mm/init.c
View File

@ -340,9 +340,7 @@ void __init arch_mm_preinit(void)
{
unsigned int flags = SWIOTLB_VERBOSE;
if (is_realm_world() || is_protected_kvm_guest()) {
flags |= SWIOTLB_FORCE;
} else if (max_pfn <= PFN_DOWN(arm64_dma_phys_limit)) {
if (max_pfn <= PFN_DOWN(arm64_dma_phys_limit)) {
/*
* If no bouncing needed for ZONE_DMA, reduce the swiotlb
* buffer for kmalloc() bouncing to 1MB per 1GB of RAM.
@ -419,6 +417,7 @@ bool cc_platform_has(enum cc_attr attr)
{
switch (attr) {
case CC_ATTR_MEM_ENCRYPT:
case CC_ATTR_GUEST_MEM_ENCRYPT:
return is_realm_world() || is_protected_kvm_guest();
default:
return false;

1
arch/powerpc/platforms/pseries/cc_platform.c
View File

@ -17,6 +17,7 @@ bool cc_platform_has(enum cc_attr attr)
{
switch (attr) {
case CC_ATTR_MEM_ENCRYPT:
case CC_ATTR_GUEST_MEM_ENCRYPT:
return is_secure_guest();
default:

2
arch/powerpc/platforms/pseries/svm.c
View File

@ -29,7 +29,7 @@ static int __init init_svm(void)
* need to use the SWIOTLB buffer for DMA even if dma_capable() says
* otherwise.
*/
ppc_swiotlb_flags |= SWIOTLB_ANY | SWIOTLB_FORCE;
ppc_swiotlb_flags |= SWIOTLB_ANY;
/* Share the SWIOTLB buffer with the host. */
swiotlb_update_mem_attributes();

1
arch/s390/Kconfig
View File

@ -87,6 +87,7 @@ config S390
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
select ARCH_HAS_PMD_SOFTLEAVES if TRANSPARENT_HUGEPAGE
select ARCH_HAS_CC_CAN_LINK
select ARCH_HAS_CC_PLATFORM
select ARCH_HAS_CPU_FINALIZE_INIT
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL

17
arch/s390/mm/init.c
View File

@ -50,6 +50,7 @@
#include <linux/virtio_anchor.h>
#include <linux/virtio_config.h>
#include <linux/execmem.h>
#include <linux/cc_platform.h>
pgd_t swapper_pg_dir[PTRS_PER_PGD] __section(".bss..swapper_pg_dir");
pgd_t invalid_pg_dir[PTRS_PER_PGD] __section(".bss..invalid_pg_dir");
@ -142,6 +143,20 @@ bool force_dma_unencrypted(struct device *dev)
return is_prot_virt_guest();
}
bool cc_platform_has(enum cc_attr attr)
{
switch (attr) {
case CC_ATTR_MEM_ENCRYPT:
case CC_ATTR_GUEST_MEM_ENCRYPT:
return is_prot_virt_guest();
default:
return false;
}
}
EXPORT_SYMBOL_GPL(cc_platform_has);
/* protected virtualization */
static void __init pv_init(void)
{
@ -151,7 +166,7 @@ static void __init pv_init(void)
virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc);
/* make sure bounce buffers are shared */
swiotlb_init(true, SWIOTLB_FORCE | SWIOTLB_VERBOSE | SWIOTLB_ANY);
swiotlb_init(true, SWIOTLB_VERBOSE | SWIOTLB_ANY);
swiotlb_update_mem_attributes();
}

30
arch/x86/kernel/amd_gart_64.c
View File

@ -180,22 +180,23 @@ static void iommu_full(struct device *dev, size_t size, int dir)
}
static inline int
need_iommu(struct device *dev, unsigned long addr, size_t size)
need_iommu(struct device *dev, unsigned long addr, size_t size, unsigned long attrs)
{
return force_iommu || !dma_capable(dev, addr, size, true);
return force_iommu || !dma_capable(dev, addr, size, true, attrs);
}
static inline int
nonforced_iommu(struct device *dev, unsigned long addr, size_t size)
nonforced_iommu(struct device *dev, unsigned long addr, size_t size,
unsigned long attrs)
{
return !dma_capable(dev, addr, size, true);
return !dma_capable(dev, addr, size, true, attrs);
}
/* Map a single continuous physical area into the IOMMU.
* Caller needs to check if the iommu is needed and flush.
*/
static dma_addr_t dma_map_area(struct device *dev, dma_addr_t phys_mem,
size_t size, int dir, unsigned long align_mask)
size_t size, int dir, unsigned long align_mask, unsigned long attrs)
{
unsigned long npages = iommu_num_pages(phys_mem, size, PAGE_SIZE);
unsigned long iommu_page;
@ -206,7 +207,7 @@ static dma_addr_t dma_map_area(struct device *dev, dma_addr_t phys_mem,
iommu_page = alloc_iommu(dev, npages, align_mask);
if (iommu_page == -1) {
if (!nonforced_iommu(dev, phys_mem, size))
if (!nonforced_iommu(dev, phys_mem, size, attrs))
return phys_mem;
if (panic_on_overflow)
panic("dma_map_area overflow %lu bytes\n", size);
@ -231,10 +232,10 @@ static dma_addr_t gart_map_phys(struct device *dev, phys_addr_t paddr,
if (unlikely(attrs & DMA_ATTR_MMIO))
return DMA_MAPPING_ERROR;
if (!need_iommu(dev, paddr, size))
if (!need_iommu(dev, paddr, size, attrs))
return paddr;
bus = dma_map_area(dev, paddr, size, dir, 0);
bus = dma_map_area(dev, paddr, size, dir, 0, attrs);
flush_gart();
return bus;
@ -289,7 +290,7 @@ static void gart_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,
/* Fallback for dma_map_sg in case of overflow */
static int dma_map_sg_nonforce(struct device *dev, struct scatterlist *sg,
int nents, int dir)
int nents, int dir, unsigned long attrs)
{
struct scatterlist *s;
int i;
@ -301,8 +302,8 @@ static int dma_map_sg_nonforce(struct device *dev, struct scatterlist *sg,
for_each_sg(sg, s, nents, i) {
unsigned long addr = sg_phys(s);
if (nonforced_iommu(dev, addr, s->length)) {
addr = dma_map_area(dev, addr, s->length, dir, 0);
if (nonforced_iommu(dev, addr, s->length, attrs)) {
addr = dma_map_area(dev, addr, s->length, dir, 0, attrs);
if (addr == DMA_MAPPING_ERROR) {
if (i > 0)
gart_unmap_sg(dev, sg, i, dir, 0);
@ -401,7 +402,7 @@ static int gart_map_sg(struct device *dev, struct scatterlist *sg, int nents,
s->dma_address = addr;
BUG_ON(s->length == 0);
nextneed = need_iommu(dev, addr, s->length);
nextneed = need_iommu(dev, addr, s->length, attrs);
/* Handle the previous not yet processed entries */
if (i > start) {
@ -449,7 +450,7 @@ static int gart_map_sg(struct device *dev, struct scatterlist *sg, int nents,
/* When it was forced or merged try again in a dumb way */
if (force_iommu || iommu_merge) {
out = dma_map_sg_nonforce(dev, sg, nents, dir);
out = dma_map_sg_nonforce(dev, sg, nents, dir, attrs);
if (out > 0)
return out;
}
@ -473,7 +474,8 @@ gart_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_addr,
return vaddr;
*dma_addr = dma_map_area(dev, virt_to_phys(vaddr), size,
DMA_BIDIRECTIONAL, (1UL << get_order(size)) - 1);
DMA_BIDIRECTIONAL,
(1UL << get_order(size)) - 1, attrs);
flush_gart();
if (unlikely(*dma_addr == DMA_MAPPING_ERROR))
goto out_free;

4
arch/x86/kernel/pci-dma.c
View File

@ -59,10 +59,8 @@ static void __init pci_swiotlb_detect(void)
* bounce buffers as the hypervisor can't access arbitrary VM memory
* that is not explicitly shared with it.
*/
if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
x86_swiotlb_enable = true;
x86_swiotlb_flags |= SWIOTLB_FORCE;
}
}
#else
static inline void __init pci_swiotlb_detect(void)

89
drivers/iommu/dma-iommu.c
View File

@ -1180,7 +1180,7 @@ static phys_addr_t iommu_dma_map_swiotlb(struct device *dev, phys_addr_t phys,
trace_swiotlb_bounced(dev, phys, size);
phys = swiotlb_tbl_map_single(dev, phys, size, iova_mask(iovad), dir,
attrs);
&attrs);
/*
* Untrusted devices should not see padding areas with random leftover
@ -1660,9 +1660,14 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
{
bool coherent = dev_is_dma_coherent(dev);
int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs);
bool is_alloc_cc_shared = attrs & __DMA_ATTR_ALLOC_CC_SHARED;
struct page *page = NULL;
void *cpu_addr;
/* Not yet supported */
if (is_alloc_cc_shared)
return NULL;
gfp |= __GFP_ZERO;
if (gfpflags_allow_blocking(gfp) &&
@ -1671,13 +1676,16 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
}
if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) &&
!gfpflags_allow_blocking(gfp) && !coherent)
!gfpflags_allow_blocking(gfp) && !coherent) {
page = dma_alloc_from_pool(dev, PAGE_ALIGN(size), &cpu_addr,
gfp, NULL);
else
gfp, attrs, NULL);
if (!page)
return NULL;
} else {
cpu_addr = iommu_dma_alloc_pages(dev, size, &page, gfp, attrs);
if (!cpu_addr)
return NULL;
if (!cpu_addr)
return NULL;
}
*handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot,
dev->coherent_dma_mask);
@ -2068,38 +2076,6 @@ static void iommu_dma_iova_unlink_range_slow(struct device *dev,
arch_sync_dma_flush();
}
static void __iommu_dma_iova_unlink(struct device *dev,
struct dma_iova_state *state, size_t offset, size_t size,
enum dma_data_direction dir, unsigned long attrs,
bool free_iova)
{
struct iommu_domain *domain = iommu_get_dma_domain(dev);
struct iommu_dma_cookie *cookie = domain->iova_cookie;
struct iova_domain *iovad = &cookie->iovad;
dma_addr_t addr = state->addr + offset;
size_t iova_start_pad = iova_offset(iovad, addr);
struct iommu_iotlb_gather iotlb_gather;
size_t unmapped;
if ((state->__size & DMA_IOVA_USE_SWIOTLB) ||
(!dev_is_dma_coherent(dev) &&
!(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))))
iommu_dma_iova_unlink_range_slow(dev, addr, size, dir, attrs);
iommu_iotlb_gather_init(&iotlb_gather);
iotlb_gather.queued = free_iova && READ_ONCE(cookie->fq_domain);
size = iova_align(iovad, size + iova_start_pad);
addr -= iova_start_pad;
unmapped = iommu_unmap_fast(domain, addr, size, &iotlb_gather);
WARN_ON(unmapped != size);
if (!iotlb_gather.queued)
iommu_iotlb_sync(domain, &iotlb_gather);
if (free_iova)
iommu_dma_free_iova(domain, addr, size, &iotlb_gather);
}
/**
* dma_iova_unlink - Unlink a range of IOVA space
* @dev: DMA device
@ -2115,7 +2091,27 @@ void dma_iova_unlink(struct device *dev, struct dma_iova_state *state,
size_t offset, size_t size, enum dma_data_direction dir,
unsigned long attrs)
{
__iommu_dma_iova_unlink(dev, state, offset, size, dir, attrs, false);
struct iommu_domain *domain = iommu_get_dma_domain(dev);
struct iommu_dma_cookie *cookie = domain->iova_cookie;
struct iova_domain *iovad = &cookie->iovad;
dma_addr_t addr = state->addr + offset;
size_t iova_start_pad = iova_offset(iovad, addr);
struct iommu_iotlb_gather iotlb_gather;
size_t unmapped;
if ((state->__size & DMA_IOVA_USE_SWIOTLB) ||
(!dev_is_dma_coherent(dev) &&
!(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))))
iommu_dma_iova_unlink_range_slow(dev, addr, size, dir, attrs);
iommu_iotlb_gather_init(&iotlb_gather);
size = iova_align(iovad, size + iova_start_pad);
addr -= iova_start_pad;
unmapped = iommu_unmap_fast(domain, addr, size, &iotlb_gather);
WARN_ON(unmapped != size);
iommu_iotlb_sync(domain, &iotlb_gather);
}
EXPORT_SYMBOL_GPL(dma_iova_unlink);
@ -2136,14 +2132,13 @@ void dma_iova_destroy(struct device *dev, struct dma_iova_state *state,
unsigned long attrs)
{
if (mapped_len)
__iommu_dma_iova_unlink(dev, state, 0, mapped_len, dir, attrs,
true);
else
/*
* We can be here if first call to dma_iova_link() failed and
* there is nothing to unlink, so let's be more clear.
*/
dma_iova_free(dev, state);
dma_iova_unlink(dev, state, 0, mapped_len, dir, attrs);
/*
* We can be here if the first call to dma_iova_link() failed and
* there is nothing to unlink, so let's be more clear.
*/
dma_iova_free(dev, state);
}
EXPORT_SYMBOL_GPL(dma_iova_destroy);

8
drivers/xen/swiotlb-xen.c
View File

@ -212,7 +212,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *dev, phys_addr_t phys,
BUG_ON(dir == DMA_NONE);
if (attrs & DMA_ATTR_MMIO) {
if (unlikely(!dma_capable(dev, phys, size, false))) {
if (unlikely(!dma_capable(dev, phys, size, false, attrs))) {
dev_err_once(
dev,
"DMA addr %pa+%zu overflow (mask %llx, bus limit %llx).\n",
@ -231,7 +231,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *dev, phys_addr_t phys,
* we can safely return the device addr and not worry about bounce
* buffering it.
*/
if (dma_capable(dev, dev_addr, size, true) &&
if (dma_capable(dev, dev_addr, size, true, attrs) &&
!dma_kmalloc_needs_bounce(dev, size, dir) &&
!range_straddles_page_boundary(phys, size) &&
!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
@ -243,7 +243,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *dev, phys_addr_t phys,
*/
trace_swiotlb_bounced(dev, dev_addr, size);
map = swiotlb_tbl_map_single(dev, phys, size, 0, dir, attrs);
map = swiotlb_tbl_map_single(dev, phys, size, 0, dir, &attrs);
if (map == (phys_addr_t)DMA_MAPPING_ERROR)
return DMA_MAPPING_ERROR;
@ -253,7 +253,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *dev, phys_addr_t phys,
/*
* Ensure that the address returned is DMA'ble
*/
if (unlikely(!dma_capable(dev, dev_addr, size, true))) {
if (unlikely(!dma_capable(dev, dev_addr, size, true, attrs))) {
__swiotlb_tbl_unmap_single(dev, map, size, dir,
attrs | DMA_ATTR_SKIP_CPU_SYNC,
swiotlb_find_pool(dev, map));

20
include/linux/dma-direct.h
View File

@ -77,6 +77,10 @@ static inline dma_addr_t dma_range_map_max(const struct bus_dma_region *map)
#ifndef phys_to_dma_unencrypted
#define phys_to_dma_unencrypted phys_to_dma
#endif
#ifndef phys_to_dma_encrypted
#define phys_to_dma_encrypted phys_to_dma
#endif
#else
static inline dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t paddr)
{
@ -90,6 +94,12 @@ static inline dma_addr_t phys_to_dma_unencrypted(struct device *dev,
{
return dma_addr_unencrypted(__phys_to_dma(dev, paddr));
}
static inline dma_addr_t phys_to_dma_encrypted(struct device *dev,
phys_addr_t paddr)
{
return dma_addr_encrypted(__phys_to_dma(dev, paddr));
}
/*
* If memory encryption is supported, phys_to_dma will set the memory encryption
* bit in the DMA address, and dma_to_phys will clear it.
@ -125,12 +135,20 @@ static inline bool force_dma_unencrypted(struct device *dev)
#endif /* CONFIG_ARCH_HAS_FORCE_DMA_UNENCRYPTED */
static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size,
bool is_ram)
bool is_ram, unsigned long attrs)
{
dma_addr_t end = addr + size - 1;
if (addr == DMA_MAPPING_ERROR)
return false;
/*
* The DMA address was derived from encrypted RAM, but this device
* requires unencrypted DMA addresses. Treat it as not DMA-capable
* so the caller can fall back to a suitable SWIOTLB pool.
*/
if (!(attrs & DMA_ATTR_CC_SHARED) && force_dma_unencrypted(dev))
return false;
if (is_ram && !IS_ENABLED(CONFIG_ARCH_DMA_ADDR_T_64BIT) &&
min(addr, end) < phys_to_dma(dev, PFN_PHYS(min_low_pfn)))
return false;

3
include/linux/dma-map-ops.h
View File

@ -212,9 +212,10 @@ void *dma_common_pages_remap(struct page **pages, size_t size, pgprot_t prot,
void dma_common_free_remap(void *cpu_addr, size_t size);
struct page *dma_alloc_from_pool(struct device *dev, size_t size,
void **cpu_addr, gfp_t flags,
void **cpu_addr, gfp_t flags, unsigned long attrs,
bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t));
bool dma_free_from_pool(struct device *dev, void *start, size_t size);
bool dma_free_from_pool_page(struct device *dev, struct page *page, size_t size);
int dma_direct_set_offset(struct device *dev, phys_addr_t cpu_start,
dma_addr_t dma_start, u64 size);

8
include/linux/dma-mapping.h
View File

@ -103,6 +103,14 @@
*/
#define DMA_ATTR_CC_SHARED (1UL << 13)
/*
* __DMA_ATTR_ALLOC_CC_SHARED: Internal DMA-mapping attribute used by
* allocation paths that create shared (decrypted) backing pages for
* confidential computing guests. Drivers must not pass this attribute to
* dma_alloc_attrs().
*/
#define __DMA_ATTR_ALLOC_CC_SHARED (1UL << 14)
/*
* A dma_addr_t can hold any valid DMA or bus address for the platform. It can
* be given to a device to use as a DMA source or target. It is specific to a

41
include/linux/swiotlb.h
View File

@ -15,8 +15,7 @@ struct page;
struct scatterlist;
#define SWIOTLB_VERBOSE (1 << 0) /* verbose initialization */
#define SWIOTLB_FORCE (1 << 1) /* force bounce buffering */
#define SWIOTLB_ANY (1 << 2) /* allow any memory for the buffer */
#define SWIOTLB_ANY (1 << 1) /* allow any memory for the buffer */
/*
* Maximum allowable number of contiguous slabs to map,
@ -32,8 +31,12 @@ struct scatterlist;
#define IO_TLB_SHIFT 11
#define IO_TLB_SIZE (1 << IO_TLB_SHIFT)
/* default to 64MB */
#define IO_TLB_DEFAULT_SIZE (64UL<<20)
/* compile-time default; overridable via CONFIG_SWIOTLB_DEFAULT_SIZE_MB */
#ifdef CONFIG_SWIOTLB
#define IO_TLB_DEFAULT_SIZE ((unsigned long)CONFIG_SWIOTLB_DEFAULT_SIZE_MB << 20)
#else
#define IO_TLB_DEFAULT_SIZE (64UL << 20)
#endif
unsigned long swiotlb_size_or_default(void);
void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,
@ -64,8 +67,9 @@ extern void __init swiotlb_update_mem_attributes(void);
* @areas: Array of memory area descriptors.
* @slots: Array of slot descriptors.
* @node: Member of the IO TLB memory pool list.
* @rcu: RCU head for swiotlb_dyn_free().
* @dyn_free: RCU work item used to free the pool from process context.
* @transient: %true if transient memory pool.
* @cc_shared: %true if the pool memory is shared for confidential computing.
*/
struct io_tlb_pool {
phys_addr_t start;
@ -79,8 +83,9 @@ struct io_tlb_pool {
struct io_tlb_slot *slots;
#ifdef CONFIG_SWIOTLB_DYNAMIC
struct list_head node;
struct rcu_head rcu;
struct rcu_work dyn_free;
bool transient;
bool cc_shared;
#endif
};
@ -92,16 +97,17 @@ struct io_tlb_pool {
* @debugfs: The dentry to debugfs.
* @force_bounce: %true if swiotlb bouncing is forced
* @for_alloc: %true if the pool is used for memory allocation
* @cc_shared: %true if the pool memory is shared for confidential computing.
* @can_grow: %true if more pools can be allocated dynamically.
* @phys_limit: Maximum allowed physical address.
* @lock: Lock to synchronize changes to the list.
* @pools: List of IO TLB memory pool descriptors (if dynamic).
* @dyn_alloc: Dynamic IO TLB pool allocation work.
* @total_used: The total number of slots in the pool that are currently used
* across all areas. Used only for calculating used_hiwater in
* debugfs.
* @used_hiwater: The high water mark for total_used. Used only for reporting
* in debugfs.
* across all areas. Used only for calculating used_hiwater via boot
* parameter swiotlb=track_hiwater and exposed via debugfs.
* @used_hiwater: The high water mark for total_used. Can be enabled at boot
* time via swiotlb=track_hiwater and exposed via debugfs.
* @transient_nslabs: The total number of slots in all transient pools that
* are currently used across all areas.
*/
@ -111,6 +117,7 @@ struct io_tlb_mem {
struct dentry *debugfs;
bool force_bounce;
bool for_alloc;
bool cc_shared;
#ifdef CONFIG_SWIOTLB_DYNAMIC
bool can_grow;
u64 phys_limit;
@ -238,7 +245,7 @@ static inline phys_addr_t default_swiotlb_limit(void)
phys_addr_t swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys,
size_t mapping_size, unsigned int alloc_aligned_mask,
enum dma_data_direction dir, unsigned long attrs);
enum dma_data_direction dir, unsigned long *attrs);
dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys,
size_t size, enum dma_data_direction dir, unsigned long attrs);
@ -282,15 +289,19 @@ static inline void swiotlb_sync_single_for_cpu(struct device *dev,
extern void swiotlb_print_info(void);
#ifdef CONFIG_DMA_RESTRICTED_POOL
struct page *swiotlb_alloc(struct device *dev, size_t size);
struct page *swiotlb_alloc(struct device *dev, size_t size,
unsigned long attrs);
bool swiotlb_free(struct device *dev, struct page *page, size_t size);
void swiotlb_free_from_pool(struct device *dev,
phys_addr_t tlb_addr, struct io_tlb_pool *pool);
static inline bool is_swiotlb_for_alloc(struct device *dev)
{
return dev->dma_io_tlb_mem->for_alloc;
}
#else
static inline struct page *swiotlb_alloc(struct device *dev, size_t size)
static inline struct page *swiotlb_alloc(struct device *dev, size_t size,
unsigned long attrs)
{
return NULL;
}
@ -299,6 +310,10 @@ static inline bool swiotlb_free(struct device *dev, struct page *page,
{
return false;
}
static inline void swiotlb_free_from_pool(struct device *dev,
phys_addr_t tlb_addr, struct io_tlb_pool *pool)
{
}
static inline bool is_swiotlb_for_alloc(struct device *dev)
{
return false;

3
include/trace/events/dma.h
View File

@ -35,7 +35,8 @@ TRACE_DEFINE_ENUM(DMA_NONE);
{ DMA_ATTR_MMIO, "MMIO" }, \
{ DMA_ATTR_DEBUGGING_IGNORE_CACHELINES, "CACHELINES_OVERLAP" }, \
{ DMA_ATTR_REQUIRE_COHERENT, "REQUIRE_COHERENT" }, \
{ DMA_ATTR_CC_SHARED, "CC_SHARED" })
{ DMA_ATTR_CC_SHARED, "CC_SHARED" }, \
{ __DMA_ATTR_ALLOC_CC_SHARED, "ALLOC_CC_SHARED" })
DECLARE_EVENT_CLASS(dma_map,
TP_PROTO(struct device *dev, phys_addr_t phys_addr, dma_addr_t dma_addr,

22
kernel/dma/Kconfig
View File

@ -86,6 +86,28 @@ config SWIOTLB
bool
select NEED_DMA_MAP_STATE
config SWIOTLB_DEFAULT_SIZE_MB
int "Default SWIOTLB bounce buffer size in MB"
depends on SWIOTLB
range 1 64
default 64
help
Sets the default size of the software IO TLB (SWIOTLB) bounce buffer
pool allocated at boot time. The default is 64 MB.
On memory-constrained embedded or mobile platforms (e.g., those with
a hardware IOMMU such as ARM SMMU covering most DMA-capable devices),
a smaller value such as 4 or 8 MB may be sufficient. The SWIOTLB is
then only needed for devices that bypass the IOMMU or have restricted
DMA address ranges.
The minimum allowed value is 1 MB. This compile-time default can be
overridden at runtime using the "swiotlb=<nslabs>" kernel command line
parameter. Refer to Documentation/admin-guide/kernel-parameters.txt
for details.
If unsure, leave at the default value of 64.
config SWIOTLB_DYNAMIC
bool "Dynamic allocation of DMA bounce buffers"
default n

10
kernel/dma/coherent.c
View File

@ -28,7 +28,7 @@ static inline struct dma_coherent_mem *dev_get_coherent_memory(struct device *de
}
static inline dma_addr_t dma_get_device_base(struct device *dev,
struct dma_coherent_mem * mem)
struct dma_coherent_mem *mem)
{
if (mem->use_dev_dma_pfn_offset)
return phys_to_dma(dev, PFN_PHYS(mem->pfn_base));
@ -69,8 +69,8 @@ static struct dma_coherent_mem *dma_init_coherent_memory(phys_addr_t phys_addr,
kfree(dma_mem);
out_unmap_membase:
memunmap(mem_base);
pr_err("Reserved memory: failed to init DMA memory pool at %pa, size %zd MiB\n",
&phys_addr, size / SZ_1M);
pr_err("Reserved memory: failed to init DMA memory pool at %pa, size %zu KiB\n",
&phys_addr, size / SZ_1K);
return ERR_PTR(-ENOMEM);
}
@ -385,8 +385,8 @@ static int __init rmem_dma_setup(unsigned long node, struct reserved_mem *rmem)
}
#endif
pr_info("Reserved memory: created DMA memory pool at %pa, size %ld MiB\n",
&rmem->base, (unsigned long)rmem->size / SZ_1M);
pr_info("Reserved memory: created DMA memory pool at %pa, size %llu KiB\n",
&rmem->base, (unsigned long long)(rmem->size / SZ_1K));
return 0;
}

331
kernel/dma/direct.c
View File

@ -14,6 +14,8 @@
#include <linux/set_memory.h>
#include <linux/slab.h>
#include <linux/pci-p2pdma.h>
#include <linux/cc_platform.h>
#include "direct.h"
/*
@ -24,11 +26,11 @@
u64 zone_dma_limit __ro_after_init = DMA_BIT_MASK(24);
static inline dma_addr_t phys_to_dma_direct(struct device *dev,
phys_addr_t phys)
phys_addr_t phys, bool unencrypted)
{
if (force_dma_unencrypted(dev))
if (unencrypted)
return phys_to_dma_unencrypted(dev, phys);
return phys_to_dma(dev, phys);
return phys_to_dma_encrypted(dev, phys);
}
static inline struct page *dma_direct_to_page(struct device *dev,
@ -39,8 +41,9 @@ static inline struct page *dma_direct_to_page(struct device *dev,
u64 dma_direct_get_required_mask(struct device *dev)
{
bool require_decrypted = force_dma_unencrypted(dev);
phys_addr_t phys = ((phys_addr_t)max_pfn << PAGE_SHIFT) - 1;
u64 max_dma = phys_to_dma_direct(dev, phys);
u64 max_dma = phys_to_dma_direct(dev, phys, require_decrypted);
return (1ULL << (fls64(max_dma) - 1)) * 2 - 1;
}
@ -69,7 +72,8 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *dev, u64 *phys_limit)
bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)
{
dma_addr_t dma_addr = phys_to_dma_direct(dev, phys);
bool require_decrypted = force_dma_unencrypted(dev);
dma_addr_t dma_addr = phys_to_dma_direct(dev, phys, require_decrypted);
if (dma_addr == DMA_MAPPING_ERROR)
return false;
@ -79,34 +83,28 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size)
static int dma_set_decrypted(struct device *dev, void *vaddr, size_t size)
{
if (!force_dma_unencrypted(dev))
return 0;
return set_memory_decrypted((unsigned long)vaddr, PFN_UP(size));
int ret;
ret = set_memory_decrypted((unsigned long)vaddr, PFN_UP(size));
if (ret)
pr_warn_ratelimited("leaking DMA memory that can't be decrypted\n");
return ret;
}
static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size)
{
int ret;
if (!force_dma_unencrypted(dev))
return 0;
ret = set_memory_encrypted((unsigned long)vaddr, PFN_UP(size));
if (ret)
pr_warn_ratelimited("leaking DMA memory that can't be re-encrypted\n");
return ret;
}
static void __dma_direct_free_pages(struct device *dev, struct page *page,
size_t size)
static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t size,
unsigned long attrs)
{
if (swiotlb_free(dev, page, size))
return;
dma_free_contiguous(dev, page, size);
}
static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t size)
{
struct page *page = swiotlb_alloc(dev, size);
struct page *page = swiotlb_alloc(dev, size, attrs);
if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) {
swiotlb_free(dev, page, size);
@ -125,9 +123,6 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size,
WARN_ON_ONCE(!PAGE_ALIGNED(size));
if (is_swiotlb_for_alloc(dev))
return dma_direct_alloc_swiotlb(dev, size);
gfp |= dma_direct_optimal_gfp_mask(dev, &phys_limit);
page = dma_alloc_contiguous(dev, size, gfp);
if (page) {
@ -164,22 +159,24 @@ static bool dma_direct_use_pool(struct device *dev, gfp_t gfp)
return !gfpflags_allow_blocking(gfp) && !is_swiotlb_for_alloc(dev);
}
static void *dma_direct_alloc_from_pool(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp)
static struct page *dma_direct_alloc_from_pool(struct device *dev, size_t size,
dma_addr_t *dma_handle, void **cpu_addr, gfp_t gfp,
unsigned long attrs)
{
struct page *page;
u64 phys_limit;
void *ret;
if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)))
return NULL;
gfp |= dma_direct_optimal_gfp_mask(dev, &phys_limit);
page = dma_alloc_from_pool(dev, size, &ret, gfp, dma_coherent_ok);
page = dma_alloc_from_pool(dev, size, cpu_addr, gfp, attrs,
dma_coherent_ok);
if (!page)
return NULL;
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));
return ret;
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page),
attrs & __DMA_ATTR_ALLOC_CC_SHARED);
return page;
}
static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,
@ -194,9 +191,11 @@ static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,
/* remove any dirty cache lines on the kernel alias */
if (!PageHighMem(page))
arch_dma_prep_coherent(page, size);
/* return the page pointer as the opaque cookie */
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));
/*
* return the page pointer as the opaque cookie.
* Never used for unencrypted allocation
*/
*dma_handle = phys_to_dma_encrypted(dev, page_to_phys(page));
return page;
}
@ -204,15 +203,31 @@ void *dma_direct_alloc(struct device *dev, size_t size,
dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs)
{
bool remap = false, set_uncached = false;
bool mark_mem_decrypt = false;
bool allow_highmem = true;
struct page *page;
void *ret;
void *cpu_addr;
if (force_dma_unencrypted(dev))
attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
if (attrs & __DMA_ATTR_ALLOC_CC_SHARED) {
/*
* Unencrypted/shared DMA requires a linear-mapped buffer
* address to look up the PFN and set architecture-required PFN
* attributes. This is not possible with HighMem. Avoid HighMem
* allocation.
*/
allow_highmem = false;
mark_mem_decrypt = true;
}
size = PAGE_ALIGN(size);
if (attrs & DMA_ATTR_NO_WARN)
gfp |= __GFP_NOWARN;
if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) &&
!force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev))
if (((attrs & (DMA_ATTR_NO_KERNEL_MAPPING | __DMA_ATTR_ALLOC_CC_SHARED)) ==
DMA_ATTR_NO_KERNEL_MAPPING) && !is_swiotlb_for_alloc(dev))
return dma_direct_alloc_no_mapping(dev, size, dma_handle, gfp);
if (!dev_is_dma_coherent(dev)) {
@ -245,16 +260,37 @@ void *dma_direct_alloc(struct device *dev, size_t size,
/*
* Remapping or decrypting memory may block, allocate the memory from
* the atomic pools instead if we aren't allowed block.
* FIXME: With CONFIG_DMA_DIRECT_REMAP, the pool is also mapped as
* DMA-coherent (non-cacheable). We may want to create a separate pool
* dedicated to CC_SHARED atomic allocations.
*/
if ((remap || force_dma_unencrypted(dev)) &&
dma_direct_use_pool(dev, gfp))
return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp);
if ((remap || (attrs & __DMA_ATTR_ALLOC_CC_SHARED)) &&
dma_direct_use_pool(dev, gfp)) {
page = dma_direct_alloc_from_pool(dev, size,
dma_handle, &cpu_addr,
gfp, attrs);
return page ? cpu_addr : NULL;
}
if (is_swiotlb_for_alloc(dev)) {
page = dma_direct_alloc_swiotlb(dev, size, attrs);
if (page) {
/*
* swiotlb allocations comes from pool already marked
* decrypted
*/
mark_mem_decrypt = false;
goto setup_page;
}
return NULL;
}
/* we always manually zero the memory once we are done */
page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true);
page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, allow_highmem);
if (!page)
return NULL;
setup_page:
/*
* dma_alloc_contiguous can return highmem pages depending on a
* combination the cma= arguments and per-arch setup. These need to be
@ -265,43 +301,56 @@ void *dma_direct_alloc(struct device *dev, size_t size,
set_uncached = false;
}
if (mark_mem_decrypt) {
void *lm_addr;
lm_addr = page_address(page);
if (set_memory_decrypted((unsigned long)lm_addr, PFN_UP(size)))
goto out_leak_pages;
}
if (remap) {
pgprot_t prot = dma_pgprot(dev, PAGE_KERNEL, attrs);
if (force_dma_unencrypted(dev))
prot = pgprot_decrypted(prot);
/* remove any dirty cache lines on the kernel alias */
arch_dma_prep_coherent(page, size);
/* create a coherent mapping */
ret = dma_common_contiguous_remap(page, size, prot,
__builtin_return_address(0));
if (!ret)
goto out_free_pages;
cpu_addr = dma_common_contiguous_remap(page, size, prot,
__builtin_return_address(0));
if (!cpu_addr)
goto out_encrypt_pages;
} else {
ret = page_address(page);
if (dma_set_decrypted(dev, ret, size))
goto out_leak_pages;
cpu_addr = page_address(page);
}
memset(ret, 0, size);
memset(cpu_addr, 0, size);
if (set_uncached) {
void *uncached_cpu_addr;
arch_dma_prep_coherent(page, size);
ret = arch_dma_set_uncached(ret, size);
if (IS_ERR(ret))
goto out_encrypt_pages;
uncached_cpu_addr = arch_dma_set_uncached(cpu_addr, size);
if (IS_ERR(uncached_cpu_addr))
goto out_free_remap_pages;
cpu_addr = uncached_cpu_addr;
}
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));
return ret;
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page),
attrs & __DMA_ATTR_ALLOC_CC_SHARED);
return cpu_addr;
out_free_remap_pages:
if (remap)
dma_common_free_remap(cpu_addr, size);
out_encrypt_pages:
if (dma_set_encrypted(dev, page_address(page), size))
return NULL;
out_free_pages:
__dma_direct_free_pages(dev, page, size);
if (mark_mem_decrypt &&
dma_set_encrypted(dev, page_address(page), size))
goto out_leak_pages;
if (!swiotlb_free(dev, page, size))
dma_free_contiguous(dev, page, size);
return NULL;
out_leak_pages:
return NULL;
@ -310,10 +359,23 @@ void *dma_direct_alloc(struct device *dev, size_t size,
void dma_direct_free(struct device *dev, size_t size,
void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs)
{
phys_addr_t phys;
bool mark_mem_encrypted = false;
struct io_tlb_pool *swiotlb_pool;
unsigned int page_order = get_order(size);
if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) &&
!force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev)) {
/*
* If the allocation used decrypted/shared backing pages, restore
* the encryption state on free.
*/
if (force_dma_unencrypted(dev))
attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
if (attrs & __DMA_ATTR_ALLOC_CC_SHARED)
mark_mem_encrypted = true;
if (((attrs & (DMA_ATTR_NO_KERNEL_MAPPING | __DMA_ATTR_ALLOC_CC_SHARED)) ==
DMA_ATTR_NO_KERNEL_MAPPING) && !is_swiotlb_for_alloc(dev)) {
/* cpu_addr is a struct page cookie, not a kernel address */
dma_free_contiguous(dev, cpu_addr, size);
return;
@ -338,36 +400,70 @@ void dma_direct_free(struct device *dev, size_t size,
dma_free_from_pool(dev, cpu_addr, PAGE_ALIGN(size)))
return;
phys = dma_to_phys(dev, dma_addr);
swiotlb_pool = swiotlb_find_pool(dev, phys);
if (swiotlb_pool)
/* Swiotlb doesn't need a page attribute update on free */
mark_mem_encrypted = false;
if (is_vmalloc_addr(cpu_addr)) {
vunmap(cpu_addr);
} else {
if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED))
arch_dma_clear_uncached(cpu_addr, size);
if (dma_set_encrypted(dev, cpu_addr, size))
return;
}
__dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size);
if (mark_mem_encrypted) {
void *lm_addr;
lm_addr = phys_to_virt(phys);
if (set_memory_encrypted((unsigned long)lm_addr, PFN_UP(size))) {
pr_warn_ratelimited("leaking DMA memory that can't be re-encrypted\n");
return;
}
}
if (swiotlb_pool)
swiotlb_free_from_pool(dev, phys, swiotlb_pool);
else
dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size);
}
struct page *dma_direct_alloc_pages(struct device *dev, size_t size,
dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp)
{
unsigned long attrs = 0;
struct page *page;
void *ret;
void *cpu_addr;
if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp))
return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp);
if (force_dma_unencrypted(dev))
attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
if ((attrs & __DMA_ATTR_ALLOC_CC_SHARED) && dma_direct_use_pool(dev, gfp))
return dma_direct_alloc_from_pool(dev, size, dma_handle,
&cpu_addr, gfp, attrs);
if (is_swiotlb_for_alloc(dev)) {
page = dma_direct_alloc_swiotlb(dev, size, attrs);
if (!page)
return NULL;
cpu_addr = page_address(page);
goto setup_page;
}
page = __dma_direct_alloc_pages(dev, size, gfp, false);
if (!page)
return NULL;
ret = page_address(page);
if (dma_set_decrypted(dev, ret, size))
cpu_addr = page_address(page);
if ((attrs & __DMA_ATTR_ALLOC_CC_SHARED) &&
dma_set_decrypted(dev, cpu_addr, size))
goto out_leak_pages;
memset(ret, 0, size);
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));
setup_page:
memset(cpu_addr, 0, size);
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page),
attrs & __DMA_ATTR_ALLOC_CC_SHARED);
return page;
out_leak_pages:
return NULL;
@ -377,16 +473,32 @@ void dma_direct_free_pages(struct device *dev, size_t size,
struct page *page, dma_addr_t dma_addr,
enum dma_data_direction dir)
{
phys_addr_t phys;
void *vaddr = page_address(page);
struct io_tlb_pool *swiotlb_pool;
/*
* if the device had requested for an unencrypted buffer,
* convert it to encrypted on free
*/
bool mark_mem_encrypted = force_dma_unencrypted(dev);
/* If cpu_addr is not from an atomic pool, dma_free_from_pool() fails */
/* If page is not from an atomic pool, dma_free_from_pool_page() fails */
if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) &&
dma_free_from_pool(dev, vaddr, size))
dma_free_from_pool_page(dev, page, size))
return;
if (dma_set_encrypted(dev, vaddr, size))
phys = page_to_phys(page);
swiotlb_pool = swiotlb_find_pool(dev, phys);
if (swiotlb_pool)
mark_mem_encrypted = false;
if (mark_mem_encrypted && dma_set_encrypted(dev, vaddr, size))
return;
__dma_direct_free_pages(dev, page, size);
if (swiotlb_pool)
swiotlb_free_from_pool(dev, phys, swiotlb_pool);
else
dma_free_contiguous(dev, page, size);
}
#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
@ -489,9 +601,8 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents,
case PCI_P2PDMA_MAP_BUS_ADDR:
sg->dma_address = pci_p2pdma_bus_addr_map(
p2pdma_state.mem, sg_phys(sg));
sg_dma_len(sg) = sg->length;
sg_dma_mark_bus_address(sg);
continue;
break;
default:
ret = -EREMOTEIO;
goto out_unmap;
@ -538,9 +649,10 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma,
const pgoff_t pgoff_end = vma_end_pgoff(vma);
int ret = -ENXIO;
vma->vm_page_prot = dma_pgprot(dev, vma->vm_page_prot, attrs);
if (force_dma_unencrypted(dev))
vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot);
attrs |= DMA_ATTR_CC_SHARED;
vma->vm_page_prot = dma_pgprot(dev, vma->vm_page_prot, attrs);
if (dma_mmap_from_dev_coherent(dev, vma, cpu_addr, size, &ret))
return ret;
@ -553,6 +665,63 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma,
user_count << PAGE_SHIFT, vma->vm_page_prot);
}
dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys,
size_t size, enum dma_data_direction dir,
unsigned long attrs, bool flush)
{
dma_addr_t dma_addr;
if (attrs & DMA_ATTR_MMIO) {
/*
* For host memory encryption treat MMIO memory as shared
*/
if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT))
attrs |= DMA_ATTR_CC_SHARED;
}
if (is_swiotlb_force_bounce(dev)) {
if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT))
return DMA_MAPPING_ERROR;
return swiotlb_map(dev, phys, size, dir, attrs);
}
if (attrs & DMA_ATTR_CC_SHARED)
dma_addr = phys_to_dma_unencrypted(dev, phys);
else
dma_addr = phys_to_dma_encrypted(dev, phys);
if (attrs & DMA_ATTR_MMIO) {
if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs)))
goto err_overflow;
goto dma_mapped;
}
if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) ||
dma_kmalloc_needs_bounce(dev, size, dir)) {
if (is_swiotlb_active(dev) &&
!(attrs & DMA_ATTR_REQUIRE_COHERENT))
return swiotlb_map(dev, phys, size, dir, attrs);
goto err_overflow;
}
dma_mapped:
if (!dev_is_dma_coherent(dev) &&
!(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) {
arch_sync_dma_for_device(phys, size, dir);
if (flush)
arch_sync_dma_flush();
}
return dma_addr;
err_overflow:
dev_WARN_ONCE(
dev, 1,
"DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n",
&dma_addr, size, *dev->dma_mask, dev->bus_dma_limit);
return DMA_MAPPING_ERROR;
}
int dma_direct_supported(struct device *dev, u64 mask)
{
u64 min_mask = ((u64)max_pfn << PAGE_SHIFT) - 1;
@ -628,8 +797,10 @@ size_t dma_direct_max_mapping_size(struct device *dev)
{
/* If SWIOTLB is active, use its maximum mapping size */
if (is_swiotlb_active(dev) &&
(dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev)))
(dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev) ||
force_dma_unencrypted(dev)))
return swiotlb_max_mapping_size(dev);
return SIZE_MAX;
}

56
kernel/dma/direct.h
View File

@ -17,6 +17,9 @@ bool dma_direct_can_mmap(struct device *dev);
int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma,
void *cpu_addr, dma_addr_t dma_addr, size_t size,
unsigned long attrs);
dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys,
size_t size, enum dma_data_direction dir,
unsigned long attrs, bool flush);
bool dma_direct_need_sync(struct device *dev, dma_addr_t dma_addr);
int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents,
enum dma_data_direction dir, unsigned long attrs);
@ -82,59 +85,6 @@ static inline void dma_direct_sync_single_for_cpu(struct device *dev,
swiotlb_sync_single_for_cpu(dev, paddr, size, dir);
}
static inline dma_addr_t dma_direct_map_phys(struct device *dev,
phys_addr_t phys, size_t size, enum dma_data_direction dir,
unsigned long attrs, bool flush)
{
dma_addr_t dma_addr;
if (is_swiotlb_force_bounce(dev)) {
if (!(attrs & DMA_ATTR_CC_SHARED)) {
if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT))
return DMA_MAPPING_ERROR;
return swiotlb_map(dev, phys, size, dir, attrs);
}
} else if (attrs & DMA_ATTR_CC_SHARED) {
return DMA_MAPPING_ERROR;
}
if (attrs & DMA_ATTR_MMIO) {
dma_addr = phys;
if (unlikely(!dma_capable(dev, dma_addr, size, false)))
goto err_overflow;
} else if (attrs & DMA_ATTR_CC_SHARED) {
dma_addr = phys_to_dma_unencrypted(dev, phys);
if (unlikely(!dma_capable(dev, dma_addr, size, false)))
goto err_overflow;
} else {
dma_addr = phys_to_dma(dev, phys);
if (unlikely(!dma_capable(dev, dma_addr, size, true)) ||
dma_kmalloc_needs_bounce(dev, size, dir)) {
if (is_swiotlb_active(dev) &&
!(attrs & DMA_ATTR_REQUIRE_COHERENT))
return swiotlb_map(dev, phys, size, dir, attrs);
goto err_overflow;
}
}
if (!dev_is_dma_coherent(dev) &&
!(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) {
arch_sync_dma_for_device(phys, size, dir);
if (flush)
arch_sync_dma_flush();
}
return dma_addr;
err_overflow:
dev_WARN_ONCE(
dev, 1,
"DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n",
&dma_addr, size, *dev->dma_mask, dev->bus_dma_limit);
return DMA_MAPPING_ERROR;
}
static inline void dma_direct_unmap_phys(struct device *dev, dma_addr_t addr,
size_t size, enum dma_data_direction dir, unsigned long attrs,
bool flush)

25
kernel/dma/mapping.c
View File

@ -537,13 +537,21 @@ EXPORT_SYMBOL(dma_get_sgtable_attrs);
*/
pgprot_t dma_pgprot(struct device *dev, pgprot_t prot, unsigned long attrs)
{
pgprot_t dma_prot;
if (dev_is_dma_coherent(dev))
return prot;
dma_prot = prot;
#ifdef CONFIG_ARCH_HAS_DMA_WRITE_COMBINE
if (attrs & DMA_ATTR_WRITE_COMBINE)
return pgprot_writecombine(prot);
else if (attrs & DMA_ATTR_WRITE_COMBINE)
dma_prot = pgprot_writecombine(prot);
#endif
return pgprot_dmacoherent(prot);
else
dma_prot = pgprot_dmacoherent(prot);
if (attrs & (DMA_ATTR_CC_SHARED | __DMA_ATTR_ALLOC_CC_SHARED))
return pgprot_decrypted(dma_prot);
else
return pgprot_encrypted(dma_prot);
}
#endif /* CONFIG_MMU */
@ -638,6 +646,15 @@ void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle,
if (WARN_ON_ONCE(flag & __GFP_COMP))
return NULL;
if (attrs & (DMA_ATTR_CC_SHARED | __DMA_ATTR_ALLOC_CC_SHARED)) {
trace_dma_alloc(dev, NULL, 0, size, DMA_BIDIRECTIONAL, flag,
attrs);
return NULL;
}
if (force_dma_unencrypted(dev))
attrs |= __DMA_ATTR_ALLOC_CC_SHARED;
if (dma_alloc_from_dev_coherent(dev, size, dma_handle, &cpu_addr)) {
trace_dma_alloc(dev, cpu_addr, *dma_handle, size,
DMA_BIDIRECTIONAL, flag, attrs);

237
kernel/dma/pool.c
View File

@ -12,12 +12,18 @@
#include <linux/set_memory.h>
#include <linux/slab.h>
#include <linux/workqueue.h>
#include <linux/cc_platform.h>
static struct gen_pool *atomic_pool_dma __ro_after_init;
struct dma_gen_pool {
bool cc_shared;
struct gen_pool *pool;
};
static struct dma_gen_pool atomic_pool_dma __ro_after_init;
static unsigned long pool_size_dma;
static struct gen_pool *atomic_pool_dma32 __ro_after_init;
static struct dma_gen_pool atomic_pool_dma32 __ro_after_init;
static unsigned long pool_size_dma32;
static struct gen_pool *atomic_pool_kernel __ro_after_init;
static struct dma_gen_pool atomic_pool_kernel __ro_after_init;
static unsigned long pool_size_kernel;
/* Size can be defined by the coherent_pool command line */
@ -76,13 +82,15 @@ static bool cma_in_zone(gfp_t gfp)
return true;
}
static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size,
static int atomic_pool_expand(struct dma_gen_pool *dma_pool, size_t pool_size,
gfp_t gfp)
{
unsigned int order;
struct page *page = NULL;
bool leak_pages = false;
void *addr;
int ret = -ENOMEM;
pgprot_t prot __maybe_unused;
/* Cannot allocate larger than MAX_PAGE_ORDER */
order = min(get_order(pool_size), MAX_PAGE_ORDER);
@ -101,8 +109,12 @@ static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size,
arch_dma_prep_coherent(page, pool_size);
#ifdef CONFIG_DMA_DIRECT_REMAP
addr = dma_common_contiguous_remap(page, pool_size,
pgprot_decrypted(pgprot_dmacoherent(PAGE_KERNEL)),
if (dma_pool->cc_shared)
prot = pgprot_decrypted(pgprot_dmacoherent(PAGE_KERNEL));
else
prot = pgprot_dmacoherent(PAGE_KERNEL);
addr = dma_common_contiguous_remap(page, pool_size, prot,
__builtin_return_address(0));
if (!addr)
goto free_page;
@ -113,12 +125,17 @@ static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size,
* Memory in the atomic DMA pools must be unencrypted, the pools do not
* shrink so no re-encryption occurs in dma_direct_free().
*/
ret = set_memory_decrypted((unsigned long)page_to_virt(page),
1 << order);
if (ret)
goto remove_mapping;
ret = gen_pool_add_virt(pool, (unsigned long)addr, page_to_phys(page),
pool_size, NUMA_NO_NODE);
if (dma_pool->cc_shared) {
ret = set_memory_decrypted((unsigned long)page_to_virt(page),
1 << order);
if (ret) {
leak_pages = true;
goto remove_mapping;
}
}
ret = gen_pool_add_virt(dma_pool->pool, (unsigned long)addr,
page_to_phys(page), pool_size, NUMA_NO_NODE);
if (ret)
goto encrypt_mapping;
@ -126,62 +143,67 @@ static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size,
return 0;
encrypt_mapping:
ret = set_memory_encrypted((unsigned long)page_to_virt(page),
1 << order);
if (WARN_ON_ONCE(ret)) {
/* Decrypt succeeded but encrypt failed, purposely leak */
goto out;
}
if (dma_pool->cc_shared &&
set_memory_encrypted((unsigned long)page_to_virt(page), 1 << order))
leak_pages = true;
remove_mapping:
#ifdef CONFIG_DMA_DIRECT_REMAP
dma_common_free_remap(addr, pool_size);
free_page:
__free_pages(page, order);
#endif
if (!leak_pages)
__free_pages(page, order);
out:
return ret;
}
static void atomic_pool_resize(struct gen_pool *pool, gfp_t gfp)
static void atomic_pool_resize(struct dma_gen_pool *dma_pool, gfp_t gfp)
{
if (pool && gen_pool_avail(pool) < atomic_pool_size)
atomic_pool_expand(pool, gen_pool_size(pool), gfp);
if (dma_pool->pool && gen_pool_avail(dma_pool->pool) < atomic_pool_size)
atomic_pool_expand(dma_pool, gen_pool_size(dma_pool->pool), gfp);
}
static void atomic_pool_work_fn(struct work_struct *work)
{
if (IS_ENABLED(CONFIG_ZONE_DMA))
atomic_pool_resize(atomic_pool_dma,
atomic_pool_resize(&atomic_pool_dma,
GFP_KERNEL | GFP_DMA);
if (IS_ENABLED(CONFIG_ZONE_DMA32))
atomic_pool_resize(atomic_pool_dma32,
atomic_pool_resize(&atomic_pool_dma32,
GFP_KERNEL | GFP_DMA32);
atomic_pool_resize(atomic_pool_kernel, GFP_KERNEL);
atomic_pool_resize(&atomic_pool_kernel, GFP_KERNEL);
}
static __init struct gen_pool *__dma_atomic_pool_init(size_t pool_size,
gfp_t gfp)
static __init struct dma_gen_pool *__dma_atomic_pool_init(struct dma_gen_pool *dma_pool,
size_t pool_size, gfp_t gfp)
{
struct gen_pool *pool;
int ret;
pool = gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE);
if (!pool)
dma_pool->pool = gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE);
if (!dma_pool->pool)
return NULL;
gen_pool_set_algo(pool, gen_pool_first_fit_order_align, NULL);
gen_pool_set_algo(dma_pool->pool, gen_pool_first_fit_order_align, NULL);
ret = atomic_pool_expand(pool, pool_size, gfp);
/* if platform is using memory encryption atomic pools are by default shared. */
if (cc_platform_has(CC_ATTR_MEM_ENCRYPT))
dma_pool->cc_shared = true;
else
dma_pool->cc_shared = false;
ret = atomic_pool_expand(dma_pool, pool_size, gfp);
if (ret) {
gen_pool_destroy(pool);
gen_pool_destroy(dma_pool->pool);
dma_pool->pool = NULL;
pr_err("DMA: failed to allocate %zu KiB %pGg pool for atomic allocation\n",
pool_size >> 10, &gfp);
return NULL;
}
pr_info("DMA: preallocated %zu KiB %pGg pool for atomic allocations\n",
gen_pool_size(pool) >> 10, &gfp);
return pool;
gen_pool_size(dma_pool->pool) >> 10, &gfp);
return dma_pool;
}
#ifdef CONFIG_ZONE_DMA32
@ -207,21 +229,22 @@ static int __init dma_atomic_pool_init(void)
/* All memory might be in the DMA zone(s) to begin with */
if (has_managed_zone(ZONE_NORMAL)) {
atomic_pool_kernel = __dma_atomic_pool_init(atomic_pool_size,
GFP_KERNEL);
if (!atomic_pool_kernel)
__dma_atomic_pool_init(&atomic_pool_kernel, atomic_pool_size, GFP_KERNEL);
if (!atomic_pool_kernel.pool)
ret = -ENOMEM;
}
if (has_managed_dma()) {
atomic_pool_dma = __dma_atomic_pool_init(atomic_pool_size,
GFP_KERNEL | GFP_DMA);
if (!atomic_pool_dma)
__dma_atomic_pool_init(&atomic_pool_dma, atomic_pool_size,
GFP_KERNEL | GFP_DMA);
if (!atomic_pool_dma.pool)
ret = -ENOMEM;
}
if (has_managed_dma32) {
atomic_pool_dma32 = __dma_atomic_pool_init(atomic_pool_size,
GFP_KERNEL | GFP_DMA32);
if (!atomic_pool_dma32)
__dma_atomic_pool_init(&atomic_pool_dma32, atomic_pool_size,
GFP_KERNEL | GFP_DMA32);
if (!atomic_pool_dma32.pool)
ret = -ENOMEM;
}
@ -230,19 +253,44 @@ static int __init dma_atomic_pool_init(void)
}
postcore_initcall(dma_atomic_pool_init);
static inline struct gen_pool *dma_guess_pool(struct gen_pool *prev, gfp_t gfp)
static inline struct dma_gen_pool *__dma_guess_pool(struct dma_gen_pool *first,
struct dma_gen_pool *second, struct dma_gen_pool *third)
{
if (prev == NULL) {
if (first->pool)
return first;
if (second && second->pool)
return second;
if (third && third->pool)
return third;
return NULL;
}
static inline struct dma_gen_pool *dma_guess_pool(struct dma_gen_pool *prev,
gfp_t gfp)
{
if (!prev) {
if (gfp & GFP_DMA)
return atomic_pool_dma ?: atomic_pool_dma32 ?: atomic_pool_kernel;
return __dma_guess_pool(&atomic_pool_dma,
&atomic_pool_dma32,
&atomic_pool_kernel);
if (gfp & GFP_DMA32)
return atomic_pool_dma32 ?: atomic_pool_dma ?: atomic_pool_kernel;
return atomic_pool_kernel ?: atomic_pool_dma32 ?: atomic_pool_dma;
return __dma_guess_pool(&atomic_pool_dma32,
&atomic_pool_dma,
&atomic_pool_kernel);
return __dma_guess_pool(&atomic_pool_kernel,
&atomic_pool_dma32,
&atomic_pool_dma);
}
if (prev == atomic_pool_kernel)
return atomic_pool_dma32 ? atomic_pool_dma32 : atomic_pool_dma;
if (prev == atomic_pool_dma32)
return atomic_pool_dma;
if (prev == &atomic_pool_kernel)
return __dma_guess_pool(&atomic_pool_dma32,
&atomic_pool_dma, NULL);
if (prev == &atomic_pool_dma32)
return __dma_guess_pool(&atomic_pool_dma, NULL, NULL);
return NULL;
}
@ -272,16 +320,20 @@ static struct page *__dma_alloc_from_pool(struct device *dev, size_t size,
}
struct page *dma_alloc_from_pool(struct device *dev, size_t size,
void **cpu_addr, gfp_t gfp,
void **cpu_addr, gfp_t gfp, unsigned long attrs,
bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t))
{
struct gen_pool *pool = NULL;
struct dma_gen_pool *dma_pool = NULL;
struct page *page;
bool pool_found = false;
while ((pool = dma_guess_pool(pool, gfp))) {
while ((dma_pool = dma_guess_pool(dma_pool, gfp))) {
if (dma_pool->cc_shared != !!(attrs & __DMA_ATTR_ALLOC_CC_SHARED))
continue;
pool_found = true;
page = __dma_alloc_from_pool(dev, size, pool, cpu_addr,
page = __dma_alloc_from_pool(dev, size, dma_pool->pool, cpu_addr,
phys_addr_ok);
if (page)
return page;
@ -296,14 +348,77 @@ struct page *dma_alloc_from_pool(struct device *dev, size_t size,
bool dma_free_from_pool(struct device *dev, void *start, size_t size)
{
struct gen_pool *pool = NULL;
struct dma_gen_pool *dma_pool = NULL;
while ((pool = dma_guess_pool(pool, 0))) {
if (!gen_pool_has_addr(pool, (unsigned long)start, size))
while ((dma_pool = dma_guess_pool(dma_pool, 0))) {
if (!gen_pool_has_addr(dma_pool->pool, (unsigned long)start, size))
continue;
gen_pool_free(pool, (unsigned long)start, size);
gen_pool_free(dma_pool->pool, (unsigned long)start, size);
return true;
}
return false;
}
struct dma_pool_phys_match {
phys_addr_t phys;
size_t size;
unsigned long addr;
bool found;
};
static void dma_pool_find_phys(struct gen_pool *pool, struct gen_pool_chunk *chunk,
void *data)
{
struct dma_pool_phys_match *match = data;
phys_addr_t end = match->phys + match->size - 1;
phys_addr_t chunk_end;
if (match->found)
return;
chunk_end = chunk->phys_addr + (chunk->end_addr - chunk->start_addr);
if (match->phys < chunk->phys_addr || end > chunk_end)
return;
match->addr = chunk->start_addr + (match->phys - chunk->phys_addr);
match->found = true;
}
static bool dma_free_from_pool_phys(struct dma_gen_pool *dma_pool, phys_addr_t phys,
size_t size)
{
struct dma_pool_phys_match match = {
.phys = phys,
.size = size,
};
gen_pool_for_each_chunk(dma_pool->pool, dma_pool_find_phys, &match);
if (!match.found)
return false;
gen_pool_free(dma_pool->pool, match.addr, size);
return true;
}
/*
* FIXME: We could avoid this by storing the remapped virtual address in
* struct page and using that for lookup.
*/
bool dma_free_from_pool_page(struct device *dev, struct page *page, size_t size)
{
struct dma_gen_pool *dma_pool = NULL;
phys_addr_t phys = page_to_phys(page);
if (!IS_ENABLED(CONFIG_DMA_DIRECT_REMAP))
return dma_free_from_pool(dev, page_address(page), size);
while ((dma_pool = dma_guess_pool(dma_pool, 0))) {
if (dma_free_from_pool_phys(dma_pool, phys, size))
return true;
}
return false;
}

445
kernel/dma/swiotlb.c
View File

@ -180,6 +180,74 @@ static unsigned int limit_nareas(unsigned int nareas, unsigned long nslots)
return nareas;
}
#ifdef CONFIG_DEBUG_FS
/*
* Track the total used slots with a global atomic value in order to have
* correct information to determine the high water mark.
*/
static void inc_used_and_hiwater_real(struct io_tlb_mem *mem,
unsigned int nslots)
{
unsigned long old_hiwater, new_used;
new_used = atomic_long_add_return(nslots, &mem->total_used);
old_hiwater = atomic_long_read(&mem->used_hiwater);
do {
if (new_used <= old_hiwater)
break;
} while (!atomic_long_try_cmpxchg(&mem->used_hiwater,
&old_hiwater, new_used));
}
static void dec_used_real(struct io_tlb_mem *mem, unsigned int nslots)
{
atomic_long_sub(nslots, &mem->total_used);
}
static void inc_used_and_hiwater_nop(struct io_tlb_mem *mem,
unsigned int nslots)
{
}
static void dec_used_nop(struct io_tlb_mem *mem, unsigned int nslots)
{
}
DEFINE_STATIC_CALL(swiotlb_inc_used, inc_used_and_hiwater_nop);
DEFINE_STATIC_CALL(swiotlb_dec_used, dec_used_nop);
static __always_inline void inc_used_and_hiwater(struct io_tlb_mem *mem,
unsigned int nslots)
{
static_call(swiotlb_inc_used)(mem, nslots);
}
static __always_inline void dec_used(struct io_tlb_mem *mem,
unsigned int nslots)
{
static_call(swiotlb_dec_used)(mem, nslots);
}
static bool track_hiwater_enabled __read_mostly;
#else
static __always_inline void inc_used_and_hiwater(struct io_tlb_mem *mem,
unsigned int nslots)
{
}
static __always_inline void dec_used(struct io_tlb_mem *mem,
unsigned int nslots)
{
}
#endif
/*
* The tracking of used slots high watermark can be enabled
* by appending "track_hiwater" to the swiotlb= boot parameter.
* When disabled the tracking functions are no-ops with near-zero
* overhead via static_call.
*/
static int __init
setup_io_tlb_npages(char *str)
{
@ -194,10 +262,24 @@ setup_io_tlb_npages(char *str)
swiotlb_adjust_nareas(simple_strtoul(str, &str, 0));
if (*str == ',')
++str;
if (!strcmp(str, "force"))
if (!strncmp(str, "force", 5)) {
swiotlb_force_bounce = true;
else if (!strcmp(str, "noforce"))
str += 5;
} else if (!strncmp(str, "noforce", 7)) {
swiotlb_force_disable = true;
str += 7;
}
#ifdef CONFIG_DEBUG_FS
if (*str == ',')
++str;
if (!strncmp(str, "track_hiwater", 13)) {
track_hiwater_enabled = true;
static_call_update(swiotlb_inc_used,
inc_used_and_hiwater_real);
static_call_update(swiotlb_dec_used, dec_used_real);
}
#endif
return 0;
}
@ -248,6 +330,23 @@ static inline unsigned long nr_slots(u64 val)
return DIV_ROUND_UP(val, IO_TLB_SIZE);
}
static void swiotlb_mark_pool_used(struct io_tlb_pool *pool)
{
unsigned long i;
for (i = 0; i < pool->nareas; i++) {
pool->areas[i].index = 0;
pool->areas[i].used = pool->area_nslabs;
}
for (i = 0; i < pool->nslabs; i++) {
pool->slots[i].list = 0;
pool->slots[i].orig_addr = INVALID_PHYS_ADDR;
pool->slots[i].alloc_size = 0;
pool->slots[i].pad_slots = 0;
}
}
/*
* Early SWIOTLB allocation may be too early to allow an architecture to
* perform the desired operations. This function allows the architecture to
@ -259,16 +358,35 @@ void __init swiotlb_update_mem_attributes(void)
struct io_tlb_pool *mem = &io_tlb_default_mem.defpool;
unsigned long bytes;
/*
* if platform support memory encryption, swiotlb buffers are
* shared by default.
*/
if (cc_platform_has(CC_ATTR_MEM_ENCRYPT))
io_tlb_default_mem.cc_shared = true;
else
io_tlb_default_mem.cc_shared = false;
if (!mem->nslabs || mem->late_alloc)
return;
bytes = PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT);
set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT);
if (io_tlb_default_mem.cc_shared) {
int ret;
ret = set_memory_decrypted((unsigned long)mem->vaddr,
bytes >> PAGE_SHIFT);
if (ret) {
pr_warn("Failed to decrypt default memory pool, disabling it\n");
swiotlb_mark_pool_used(mem);
}
}
}
static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t start,
unsigned long nslabs, bool late_alloc, unsigned int nareas)
void *vaddr, unsigned long nslabs, bool late_alloc,
unsigned int nareas)
{
void *vaddr = phys_to_virt(start);
unsigned long bytes = nslabs << IO_TLB_SHIFT, i;
mem->nslabs = nslabs;
@ -364,8 +482,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,
if (swiotlb_force_disable)
return;
io_tlb_default_mem.force_bounce =
swiotlb_force_bounce || (flags & SWIOTLB_FORCE);
io_tlb_default_mem.force_bounce = swiotlb_force_bounce;
#ifdef CONFIG_SWIOTLB_DYNAMIC
if (!remap)
@ -409,7 +526,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,
return;
}
swiotlb_init_io_tlb_pool(mem, __pa(tlb), nslabs, false, nareas);
swiotlb_init_io_tlb_pool(mem, __pa(tlb), tlb, nslabs, false, nareas);
add_mem_pool(&io_tlb_default_mem, mem);
if (flags & SWIOTLB_VERBOSE)
@ -431,9 +548,10 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,
{
struct io_tlb_pool *mem = &io_tlb_default_mem.defpool;
unsigned long nslabs = ALIGN(size >> IO_TLB_SHIFT, IO_TLB_SEGSIZE);
unsigned int order, area_order, slot_order;
bool leak_pages = false;
unsigned int nareas;
unsigned char *vstart = NULL;
unsigned int order, area_order;
bool retried = false;
int rc = 0;
@ -493,6 +611,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,
(PAGE_SIZE << order) >> 20);
}
rc = -ENOMEM;
nareas = limit_nareas(default_nareas, nslabs);
area_order = get_order(array_size(sizeof(*mem->areas), nareas));
mem->areas = (struct io_tlb_area *)
@ -500,30 +619,42 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask,
if (!mem->areas)
goto error_area;
slot_order = get_order(array_size(sizeof(*mem->slots), nslabs));
mem->slots = (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO,
get_order(array_size(sizeof(*mem->slots), nslabs)));
slot_order);
if (!mem->slots)
goto error_slots;
set_memory_decrypted((unsigned long)vstart,
(nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT);
swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), nslabs, true,
if (io_tlb_default_mem.cc_shared) {
rc = set_memory_decrypted((unsigned long)vstart,
(nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT);
if (rc) {
leak_pages = true;
goto error_decrypt;
}
}
swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), vstart, nslabs, true,
nareas);
add_mem_pool(&io_tlb_default_mem, mem);
swiotlb_print_info();
return 0;
error_decrypt:
free_pages((unsigned long)mem->slots, slot_order);
error_slots:
free_pages((unsigned long)mem->areas, area_order);
error_area:
free_pages((unsigned long)vstart, order);
return -ENOMEM;
if (!leak_pages)
free_pages((unsigned long)vstart, order);
return rc;
}
void __init swiotlb_exit(void)
{
struct io_tlb_pool *mem = &io_tlb_default_mem.defpool;
bool leak_pages = false;
unsigned long tbl_vaddr;
size_t tbl_size, slots_size;
unsigned int area_order;
@ -539,17 +670,23 @@ void __init swiotlb_exit(void)
tbl_size = PAGE_ALIGN(mem->end - mem->start);
slots_size = PAGE_ALIGN(array_size(sizeof(*mem->slots), mem->nslabs));
set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT);
if (io_tlb_default_mem.cc_shared) {
if (set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT))
leak_pages = true;
}
if (mem->late_alloc) {
area_order = get_order(array_size(sizeof(*mem->areas),
mem->nareas));
free_pages((unsigned long)mem->areas, area_order);
free_pages(tbl_vaddr, get_order(tbl_size));
if (!leak_pages)
free_pages(tbl_vaddr, get_order(tbl_size));
free_pages((unsigned long)mem->slots, get_order(slots_size));
} else {
memblock_free(mem->areas,
array_size(sizeof(*mem->areas), mem->nareas));
memblock_phys_free(mem->start, tbl_size);
if (!leak_pages)
memblock_phys_free(mem->start, tbl_size);
memblock_free(mem->slots, slots_size);
}
@ -563,6 +700,7 @@ void __init swiotlb_exit(void)
* @gfp: GFP flags for the allocation.
* @bytes: Size of the buffer.
* @phys_limit: Maximum allowed physical address of the buffer.
* @attrs: DMA attributes for the allocation.
*
* Allocate pages from the buddy allocator. If successful, make the allocated
* pages decrypted that they can be used for DMA.
@ -570,9 +708,11 @@ void __init swiotlb_exit(void)
* Return: Decrypted pages, %NULL on allocation failure, or ERR_PTR(-EAGAIN)
* if the allocated physical address was above @phys_limit.
*/
static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, u64 phys_limit)
static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes,
u64 phys_limit, unsigned long attrs)
{
unsigned int order = get_order(bytes);
bool cc_shared = attrs & __DMA_ATTR_ALLOC_CC_SHARED;
struct page *page;
phys_addr_t paddr;
void *vaddr;
@ -588,13 +728,13 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, u64 phys_limit)
}
vaddr = phys_to_virt(paddr);
if (set_memory_decrypted((unsigned long)vaddr, PFN_UP(bytes)))
if (cc_shared && set_memory_decrypted((unsigned long)vaddr, PFN_UP(bytes)))
goto error;
return page;
error:
/* Intentional leak if pages cannot be encrypted again. */
if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes)))
if (cc_shared && !set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes)))
__free_pages(page, order);
return NULL;
}
@ -602,29 +742,33 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, u64 phys_limit)
/**
* swiotlb_alloc_tlb() - allocate a dynamic IO TLB buffer
* @dev: Device for which a memory pool is allocated.
* @mem: SWIOTLB allocator for the pool.
* @bytes: Size of the buffer.
* @phys_limit: Maximum allowed physical address of the buffer.
* @gfp: GFP flags for the allocation.
* @vaddr: Receives the virtual address for the allocated buffer.
*
* Return: Allocated pages, or %NULL on allocation failure.
*/
static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes,
u64 phys_limit, gfp_t gfp)
static struct page *swiotlb_alloc_tlb(struct device *dev,
struct io_tlb_mem *mem, size_t bytes,
u64 phys_limit, gfp_t gfp, void **vaddr)
{
struct page *page;
unsigned long attrs = mem->cc_shared ? __DMA_ATTR_ALLOC_CC_SHARED : 0;
*vaddr = NULL;
/*
* Allocate from the atomic pools if memory is encrypted and
* the allocation is atomic, because decrypting may block.
*/
if (!gfpflags_allow_blocking(gfp) && dev && force_dma_unencrypted(dev)) {
void *vaddr;
if (!gfpflags_allow_blocking(gfp) && dev && mem->cc_shared) {
if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL))
return NULL;
return dma_alloc_from_pool(dev, bytes, &vaddr, gfp,
dma_coherent_ok);
return dma_alloc_from_pool(dev, bytes, vaddr, gfp,
attrs, dma_coherent_ok);
}
gfp &= ~GFP_ZONEMASK;
@ -633,7 +777,7 @@ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes,
else if (phys_limit <= DMA_BIT_MASK(32))
gfp |= __GFP_DMA32;
while (IS_ERR(page = alloc_dma_pages(gfp, bytes, phys_limit))) {
while (IS_ERR(page = alloc_dma_pages(gfp, bytes, phys_limit, attrs))) {
if (IS_ENABLED(CONFIG_ZONE_DMA32) &&
phys_limit < DMA_BIT_MASK(64) &&
!(gfp & (__GFP_DMA32 | __GFP_DMA)))
@ -645,6 +789,8 @@ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes,
return NULL;
}
if (page)
*vaddr = phys_to_virt(page_to_phys(page));
return page;
}
@ -652,21 +798,25 @@ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes,
* swiotlb_free_tlb() - free a dynamically allocated IO TLB buffer
* @vaddr: Virtual address of the buffer.
* @bytes: Size of the buffer.
* @cc_shared: true if @vaddr was allocated decrypted and must be
* re-encrypted before being freed
*/
static void swiotlb_free_tlb(void *vaddr, size_t bytes)
static void swiotlb_free_tlb(void *vaddr, size_t bytes, bool cc_shared)
{
if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) &&
dma_free_from_pool(NULL, vaddr, bytes))
return;
/* Intentional leak if pages cannot be encrypted again. */
if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes)))
if (!cc_shared ||
!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes)))
__free_pages(virt_to_page(vaddr), get_order(bytes));
}
/**
* swiotlb_alloc_pool() - allocate a new IO TLB memory pool
* @dev: Device for which a memory pool is allocated.
* @mem: SWIOTLB allocator for the pool.
* @minslabs: Minimum number of slabs.
* @nslabs: Desired (maximum) number of slabs.
* @nareas: Number of areas.
@ -680,11 +830,13 @@ static void swiotlb_free_tlb(void *vaddr, size_t bytes)
* Return: New memory pool, or %NULL on allocation failure.
*/
static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev,
unsigned long minslabs, unsigned long nslabs,
unsigned int nareas, u64 phys_limit, gfp_t gfp)
struct io_tlb_mem *mem, unsigned long minslabs,
unsigned long nslabs, unsigned int nareas, u64 phys_limit,
gfp_t gfp)
{
struct io_tlb_pool *pool;
unsigned int slot_order;
void *tlb_vaddr;
struct page *tlb;
size_t pool_size;
size_t tlb_size;
@ -699,9 +851,11 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev,
if (!pool)
goto error;
pool->areas = (void *)pool + sizeof(*pool);
pool->cc_shared = mem->cc_shared;
tlb_size = nslabs << IO_TLB_SHIFT;
while (!(tlb = swiotlb_alloc_tlb(dev, tlb_size, phys_limit, gfp))) {
while (!(tlb = swiotlb_alloc_tlb(dev, mem, tlb_size,
phys_limit, gfp, &tlb_vaddr))) {
if (nslabs <= minslabs)
goto error_tlb;
nslabs = ALIGN(nslabs >> 1, IO_TLB_SEGSIZE);
@ -715,11 +869,12 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev,
if (!pool->slots)
goto error_slots;
swiotlb_init_io_tlb_pool(pool, page_to_phys(tlb), nslabs, true, nareas);
swiotlb_init_io_tlb_pool(pool, page_to_phys(tlb), tlb_vaddr, nslabs,
true, nareas);
return pool;
error_slots:
swiotlb_free_tlb(page_address(tlb), tlb_size);
swiotlb_free_tlb(tlb_vaddr, tlb_size, mem->cc_shared);
error_tlb:
kfree(pool);
error:
@ -736,7 +891,7 @@ static void swiotlb_dyn_alloc(struct work_struct *work)
container_of(work, struct io_tlb_mem, dyn_alloc);
struct io_tlb_pool *pool;
pool = swiotlb_alloc_pool(NULL, IO_TLB_MIN_SLABS, default_nslabs,
pool = swiotlb_alloc_pool(NULL, mem, IO_TLB_MIN_SLABS, default_nslabs,
default_nareas, mem->phys_limit, GFP_KERNEL);
if (!pool) {
pr_warn_ratelimited("Failed to allocate new pool");
@ -746,21 +901,24 @@ static void swiotlb_dyn_alloc(struct work_struct *work)
add_mem_pool(mem, pool);
}
/**
* swiotlb_dyn_free() - RCU callback to free a memory pool
* @rcu: RCU head in the corresponding struct io_tlb_pool.
*/
static void swiotlb_dyn_free(struct rcu_head *rcu)
static void swiotlb_dyn_free_work(struct work_struct *work)
{
struct io_tlb_pool *pool = container_of(rcu, struct io_tlb_pool, rcu);
struct io_tlb_pool *pool =
container_of(to_rcu_work(work), struct io_tlb_pool, dyn_free);
size_t slots_size = array_size(sizeof(*pool->slots), pool->nslabs);
size_t tlb_size = pool->end - pool->start;
free_pages((unsigned long)pool->slots, get_order(slots_size));
swiotlb_free_tlb(pool->vaddr, tlb_size);
swiotlb_free_tlb(pool->vaddr, tlb_size, pool->cc_shared);
kfree(pool);
}
static void swiotlb_schedule_dyn_free(struct io_tlb_pool *pool)
{
INIT_RCU_WORK(&pool->dyn_free, swiotlb_dyn_free_work);
queue_rcu_work(system_wq, &pool->dyn_free);
}
/**
* __swiotlb_find_pool() - find the IO TLB pool for a physical address
* @dev: Device which has mapped the DMA buffer.
@ -807,7 +965,7 @@ static void swiotlb_del_pool(struct device *dev, struct io_tlb_pool *pool)
list_del_rcu(&pool->node);
spin_unlock_irqrestore(&dev->dma_io_tlb_lock, flags);
call_rcu(&pool->rcu, swiotlb_dyn_free);
swiotlb_schedule_dyn_free(pool);
}
#endif /* CONFIG_SWIOTLB_DYNAMIC */
@ -959,40 +1117,6 @@ static unsigned int wrap_area_index(struct io_tlb_pool *mem, unsigned int index)
return index;
}
/*
* Track the total used slots with a global atomic value in order to have
* correct information to determine the high water mark. The mem_used()
* function gives imprecise results because there's no locking across
* multiple areas.
*/
#ifdef CONFIG_DEBUG_FS
static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslots)
{
unsigned long old_hiwater, new_used;
new_used = atomic_long_add_return(nslots, &mem->total_used);
old_hiwater = atomic_long_read(&mem->used_hiwater);
do {
if (new_used <= old_hiwater)
break;
} while (!atomic_long_try_cmpxchg(&mem->used_hiwater,
&old_hiwater, new_used));
}
static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
{
atomic_long_sub(nslots, &mem->total_used);
}
#else /* !CONFIG_DEBUG_FS */
static void inc_used_and_hiwater(struct io_tlb_mem *mem, unsigned int nslots)
{
}
static void dec_used(struct io_tlb_mem *mem, unsigned int nslots)
{
}
#endif /* CONFIG_DEBUG_FS */
#ifdef CONFIG_SWIOTLB_DYNAMIC
#ifdef CONFIG_DEBUG_FS
static void inc_transient_used(struct io_tlb_mem *mem, unsigned int nslots)
@ -1021,6 +1145,7 @@ static void dec_transient_used(struct io_tlb_mem *mem, unsigned int nslots)
* @pool: Memory pool to be searched.
* @area_index: Index of the IO TLB memory area to be searched.
* @orig_addr: Original (non-bounced) IO buffer address.
* @tbl_dma_addr: DMA address of the bounce buffer.
* @alloc_size: Total requested size of the bounce buffer,
* including initial alignment padding.
* @alloc_align_mask: Required alignment of the allocated buffer.
@ -1032,13 +1157,11 @@ static void dec_transient_used(struct io_tlb_mem *mem, unsigned int nslots)
* Return: Index of the first allocated slot, or -1 on error.
*/
static int swiotlb_search_pool_area(struct device *dev, struct io_tlb_pool *pool,
int area_index, phys_addr_t orig_addr, size_t alloc_size,
unsigned int alloc_align_mask)
int area_index, phys_addr_t orig_addr, dma_addr_t tbl_dma_addr,
size_t alloc_size, unsigned int alloc_align_mask)
{
struct io_tlb_area *area = pool->areas + area_index;
unsigned long boundary_mask = dma_get_seg_boundary(dev);
dma_addr_t tbl_dma_addr =
phys_to_dma_unencrypted(dev, pool->start) & boundary_mask;
unsigned long max_slots = get_max_slots(boundary_mask);
unsigned int iotlb_align_mask = dma_get_min_align_mask(dev);
unsigned int nslots = nr_slots(alloc_size), stride;
@ -1051,6 +1174,8 @@ static int swiotlb_search_pool_area(struct device *dev, struct io_tlb_pool *pool
BUG_ON(!nslots);
BUG_ON(area_index >= pool->nareas);
tbl_dma_addr &= boundary_mask;
/*
* Historically, swiotlb allocations >= PAGE_SIZE were guaranteed to be
* page-aligned in the absence of any other alignment requirements.
@ -1162,6 +1287,7 @@ static int swiotlb_search_area(struct device *dev, int start_cpu,
{
struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
struct io_tlb_pool *pool;
dma_addr_t tbl_dma_addr;
int area_index;
int index = -1;
@ -1170,9 +1296,15 @@ static int swiotlb_search_area(struct device *dev, int start_cpu,
if (cpu_offset >= pool->nareas)
continue;
area_index = (start_cpu + cpu_offset) & (pool->nareas - 1);
if (mem->cc_shared)
tbl_dma_addr = phys_to_dma_unencrypted(dev, pool->start);
else
tbl_dma_addr = phys_to_dma_encrypted(dev, pool->start);
index = swiotlb_search_pool_area(dev, pool, area_index,
orig_addr, alloc_size,
alloc_align_mask);
orig_addr, tbl_dma_addr,
alloc_size, alloc_align_mask);
if (index >= 0) {
*retpool = pool;
break;
@ -1202,6 +1334,7 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr,
{
struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
struct io_tlb_pool *pool;
dma_addr_t tbl_dma_addr;
unsigned long nslabs;
unsigned long flags;
u64 phys_limit;
@ -1226,15 +1359,20 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr,
nslabs = nr_slots(alloc_size);
phys_limit = min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
pool = swiotlb_alloc_pool(dev, nslabs, nslabs, 1, phys_limit,
pool = swiotlb_alloc_pool(dev, mem, nslabs, nslabs, 1, phys_limit,
GFP_NOWAIT);
if (!pool)
return -1;
index = swiotlb_search_pool_area(dev, pool, 0, orig_addr,
if (mem->cc_shared)
tbl_dma_addr = phys_to_dma_unencrypted(dev, pool->start);
else
tbl_dma_addr = phys_to_dma_encrypted(dev, pool->start);
index = swiotlb_search_pool_area(dev, pool, 0, orig_addr, tbl_dma_addr,
alloc_size, alloc_align_mask);
if (index < 0) {
swiotlb_dyn_free(&pool->rcu);
swiotlb_schedule_dyn_free(pool);
return -1;
}
@ -1276,15 +1414,23 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr,
size_t alloc_size, unsigned int alloc_align_mask,
struct io_tlb_pool **retpool)
{
struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
struct io_tlb_pool *pool;
dma_addr_t tbl_dma_addr;
int start, i;
int index;
*retpool = pool = &dev->dma_io_tlb_mem->defpool;
*retpool = pool = &mem->defpool;
if (mem->cc_shared)
tbl_dma_addr = phys_to_dma_unencrypted(dev, pool->start);
else
tbl_dma_addr = phys_to_dma_encrypted(dev, pool->start);
i = start = raw_smp_processor_id() & (pool->nareas - 1);
do {
index = swiotlb_search_pool_area(dev, pool, i, orig_addr,
alloc_size, alloc_align_mask);
tbl_dma_addr, alloc_size,
alloc_align_mask);
if (index >= 0)
return index;
if (++i >= pool->nareas)
@ -1295,24 +1441,6 @@ static int swiotlb_find_slots(struct device *dev, phys_addr_t orig_addr,
#endif /* CONFIG_SWIOTLB_DYNAMIC */
#ifdef CONFIG_DEBUG_FS
/**
* mem_used() - get number of used slots in an allocator
* @mem: Software IO TLB allocator.
*
* The result is accurate in this version of the function, because an atomic
* counter is available if CONFIG_DEBUG_FS is set.
*
* Return: Number of used slots.
*/
static unsigned long mem_used(struct io_tlb_mem *mem)
{
return atomic_long_read(&mem->total_used);
}
#else /* !CONFIG_DEBUG_FS */
/**
* mem_pool_used() - get number of used slots in a memory pool
* @pool: Software IO TLB memory pool.
@ -1335,13 +1463,20 @@ static unsigned long mem_pool_used(struct io_tlb_pool *pool)
* mem_used() - get number of used slots in an allocator
* @mem: Software IO TLB allocator.
*
* The result is not accurate, because there is no locking of individual
* areas.
* When trace_hiwater and CONFIG_DEBUG_FS is enabled, the result is accurate
* because the total number of used slots is tracked in mem->total_used.
* Otherwise, the result is an approximation, because there is no locking of
* individual areas.
*
* Return: Approximate number of used slots.
* Return: Number of used slots.
*/
static unsigned long mem_used(struct io_tlb_mem *mem)
{
#ifdef CONFIG_DEBUG_FS
if (track_hiwater_enabled)
return atomic_long_read(&mem->total_used);
#endif
#ifdef CONFIG_SWIOTLB_DYNAMIC
struct io_tlb_pool *pool;
unsigned long used = 0;
@ -1357,8 +1492,6 @@ static unsigned long mem_used(struct io_tlb_mem *mem)
#endif
}
#endif /* CONFIG_DEBUG_FS */
/**
* swiotlb_tbl_map_single() - bounce buffer map a single contiguous physical area
* @dev: Device which maps the buffer.
@ -1367,9 +1500,19 @@ static unsigned long mem_used(struct io_tlb_mem *mem)
* any pre- or post-padding for alignment
* @alloc_align_mask: Required start and end alignment of the allocated buffer
* @dir: DMA direction
* @attrs: Optional DMA attributes for the map operation
* @attrs: Optional DMA attributes for the map operation, updated
* to match the selected SWIOTLB pool
*
* Find and allocate a suitable sequence of IO TLB slots for the request.
* The device's SWIOTLB pool must match the device's current DMA encryption
* requirements. If the device requires decrypted DMA, bouncing is done through
* an unencrypted pool and the mapping is marked shared. If the device can DMA
* to encrypted memory, bouncing is done through an encrypted pool even when the
* original DMA address was unencrypted. Enabling encrypted DMA for a device is
* therefore expected to update its default io_tlb_mem to an encrypted pool, so
* later bounce mappings for both encrypted and decrypted original memory use
* that encrypted pool.
*
* The allocated space starts at an alignment specified by alloc_align_mask,
* and the size of the allocated space is rounded up so that the total amount
* of allocated space is a multiple of (alloc_align_mask + 1). If
@ -1386,7 +1529,7 @@ static unsigned long mem_used(struct io_tlb_mem *mem)
*/
phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr,
size_t mapping_size, unsigned int alloc_align_mask,
enum dma_data_direction dir, unsigned long attrs)
enum dma_data_direction dir, unsigned long *attrs)
{
struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
unsigned int offset;
@ -1406,6 +1549,30 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr,
if (cc_platform_has(CC_ATTR_MEM_ENCRYPT))
pr_warn_once("Memory encryption is active and system is using DMA bounce buffers\n");
if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
/* swiotlb pool is incorrect for this device */
if (unlikely(mem->cc_shared != force_dma_unencrypted(dev)))
return (phys_addr_t)DMA_MAPPING_ERROR;
} else if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) {
/*
* On hosts with memory encryption, SWIOTLB-backed memory is
* unencrypted. DMA addresses returned for bounce buffers must
* therefore be marked unencrypted, even for devices that can
* address encrypted memory. This also preserves swiotlb=force
* behavior for those devices.
*/
if (unlikely(!mem->cc_shared))
return (phys_addr_t)DMA_MAPPING_ERROR;
}
/* Force attrs to match the kind of memory in the pool */
if (mem->cc_shared)
*attrs |= DMA_ATTR_CC_SHARED;
else
*attrs &= ~DMA_ATTR_CC_SHARED;
/*
* The default swiotlb memory pool is allocated with PAGE_SIZE
* alignment. If a mapping is requested with larger alignment,
@ -1420,7 +1587,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr,
size = ALIGN(mapping_size + offset, alloc_align_mask + 1);
index = swiotlb_find_slots(dev, orig_addr, size, alloc_align_mask, &pool);
if (index == -1) {
if (!(attrs & DMA_ATTR_NO_WARN))
if (!(*attrs & DMA_ATTR_NO_WARN))
dev_warn_ratelimited(dev,
"swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n",
size, mem->nslabs, mem_used(mem));
@ -1599,13 +1766,16 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_t paddr, size_t size,
trace_swiotlb_bounced(dev, phys_to_dma(dev, paddr), size);
swiotlb_addr = swiotlb_tbl_map_single(dev, paddr, size, 0, dir, attrs);
swiotlb_addr = swiotlb_tbl_map_single(dev, paddr, size, 0, dir, &attrs);
if (swiotlb_addr == (phys_addr_t)DMA_MAPPING_ERROR)
return DMA_MAPPING_ERROR;
/* Ensure that the address returned is DMA'ble */
dma_addr = phys_to_dma_unencrypted(dev, swiotlb_addr);
if (unlikely(!dma_capable(dev, dma_addr, size, true))) {
if (attrs & DMA_ATTR_CC_SHARED)
dma_addr = phys_to_dma_unencrypted(dev, swiotlb_addr);
else
dma_addr = phys_to_dma_encrypted(dev, swiotlb_addr);
if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs))) {
__swiotlb_tbl_unmap_single(dev, swiotlb_addr, size, dir,
attrs | DMA_ATTR_SKIP_CPU_SYNC,
swiotlb_find_pool(dev, swiotlb_addr));
@ -1768,7 +1938,7 @@ static inline void swiotlb_create_debugfs_files(struct io_tlb_mem *mem,
#ifdef CONFIG_DMA_RESTRICTED_POOL
struct page *swiotlb_alloc(struct device *dev, size_t size)
struct page *swiotlb_alloc(struct device *dev, size_t size, unsigned long attrs)
{
struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
struct io_tlb_pool *pool;
@ -1779,6 +1949,9 @@ struct page *swiotlb_alloc(struct device *dev, size_t size)
if (!mem)
return NULL;
if (mem->cc_shared != !!(attrs & __DMA_ATTR_ALLOC_CC_SHARED))
return NULL;
align = (1 << (get_order(size) + PAGE_SHIFT)) - 1;
index = swiotlb_find_slots(dev, 0, size, align, &pool);
if (index == -1)
@ -1809,6 +1982,12 @@ bool swiotlb_free(struct device *dev, struct page *page, size_t size)
return true;
}
void swiotlb_free_from_pool(struct device *dev,
phys_addr_t tlb_addr, struct io_tlb_pool *pool)
{
swiotlb_release_slots(dev, tlb_addr, pool);
}
static int rmem_swiotlb_device_init(struct reserved_mem *rmem,
struct device *dev)
{
@ -1848,11 +2027,29 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem,
kfree(mem);
return -ENOMEM;
}
/*
* if platform supports memory encryption,
* restricted mem pool is shared by default
*/
if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
int ret;
set_memory_decrypted((unsigned long)phys_to_virt(rmem->base),
rmem->size >> PAGE_SHIFT);
swiotlb_init_io_tlb_pool(pool, rmem->base, nslabs,
false, nareas);
mem->cc_shared = true;
ret = set_memory_decrypted((unsigned long)phys_to_virt(rmem->base),
rmem->size >> PAGE_SHIFT);
if (ret) {
dev_err(dev, "Failed to decrypt restricted DMA pool\n");
kfree(pool->areas);
kfree(pool->slots);
kfree(mem);
return ret;
}
} else {
mem->cc_shared = false;
}
swiotlb_init_io_tlb_pool(pool, rmem->base, phys_to_virt(rmem->base),
nslabs, false, nareas);
mem->force_bounce = true;
mem->for_alloc = true;
#ifdef CONFIG_SWIOTLB_DYNAMIC