Azure Storage File Datalake Py

1---
2name: azure-storage-file-datalake-py
3description: |
4  Azure Data Lake Storage Gen2 SDK for Python. Use for hierarchical file systems, big data analytics, and file/directory operations.
5  Triggers: "data lake", "DataLakeServiceClient", "FileSystemClient", "ADLS Gen2", "hierarchical namespace".
6package: azure-storage-file-datalake
7---
8 
9# Azure Data Lake Storage Gen2 SDK for Python
10 
11Hierarchical file system for big data analytics workloads.
12 
13## Installation
14 
15```bash
16pip install azure-storage-file-datalake azure-identity
17```
18 
19## Environment Variables
20 
21```bash
22AZURE_STORAGE_ACCOUNT_URL=https://<account>.dfs.core.windows.net
23```
24 
25## Authentication
26 
27```python
28from azure.identity import DefaultAzureCredential
29from azure.storage.filedatalake import DataLakeServiceClient
30 
31credential = DefaultAzureCredential()
32account_url = "https://<account>.dfs.core.windows.net"
33 
34service_client = DataLakeServiceClient(account_url=account_url, credential=credential)
35```
36 
37## Client Hierarchy
38 
39| Client | Purpose |
40|--------|---------|
41| `DataLakeServiceClient` | Account-level operations |
42| `FileSystemClient` | Container (file system) operations |
43| `DataLakeDirectoryClient` | Directory operations |
44| `DataLakeFileClient` | File operations |
45 
46## File System Operations
47 
48```python
49# Create file system (container)
50file_system_client = service_client.create_file_system("myfilesystem")
51 
52# Get existing
53file_system_client = service_client.get_file_system_client("myfilesystem")
54 
55# Delete
56service_client.delete_file_system("myfilesystem")
57 
58# List file systems
59for fs in service_client.list_file_systems():
60    print(fs.name)
61```
62 
63## Directory Operations
64 
65```python
66file_system_client = service_client.get_file_system_client("myfilesystem")
67 
68# Create directory
69directory_client = file_system_client.create_directory("mydir")
70 
71# Create nested directories
72directory_client = file_system_client.create_directory("path/to/nested/dir")
73 
74# Get directory client
75directory_client = file_system_client.get_directory_client("mydir")
76 
77# Delete directory
78directory_client.delete_directory()
79 
80# Rename/move directory
81directory_client.rename_directory(new_name="myfilesystem/newname")
82```
83 
84## File Operations
85 
86### Upload File
87 
88```python
89# Get file client
90file_client = file_system_client.get_file_client("path/to/file.txt")
91 
92# Upload from local file
93with open("local-file.txt", "rb") as data:
94    file_client.upload_data(data, overwrite=True)
95 
96# Upload bytes
97file_client.upload_data(b"Hello, Data Lake!", overwrite=True)
98 
99# Append data (for large files)
100file_client.append_data(data=b"chunk1", offset=0, length=6)
101file_client.append_data(data=b"chunk2", offset=6, length=6)
102file_client.flush_data(12)  # Commit the data
103```
104 
105### Download File
106 
107```python
108file_client = file_system_client.get_file_client("path/to/file.txt")
109 
110# Download all content
111download = file_client.download_file()
112content = download.readall()
113 
114# Download to file
115with open("downloaded.txt", "wb") as f:
116    download = file_client.download_file()
117    download.readinto(f)
118 
119# Download range
120download = file_client.download_file(offset=0, length=100)
121```
122 
123### Delete File
124 
125```python
126file_client.delete_file()
127```
128 
129## List Contents
130 
131```python
132# List paths (files and directories)
133for path in file_system_client.get_paths():
134    print(f"{'DIR' if path.is_directory else 'FILE'}: {path.name}")
135 
136# List paths in directory
137for path in file_system_client.get_paths(path="mydir"):
138    print(path.name)
139 
140# Recursive listing
141for path in file_system_client.get_paths(path="mydir", recursive=True):
142    print(path.name)
143```
144 
145## File/Directory Properties
146 
147```python
148# Get properties
149properties = file_client.get_file_properties()
150print(f"Size: {properties.size}")
151print(f"Last modified: {properties.last_modified}")
152 
153# Set metadata
154file_client.set_metadata(metadata={"processed": "true"})
155```
156 
157## Access Control (ACL)
158 
159```python
160# Get ACL
161acl = directory_client.get_access_control()
162print(f"Owner: {acl['owner']}")
163print(f"Permissions: {acl['permissions']}")
164 
165# Set ACL
166directory_client.set_access_control(
167    owner="user-id",
168    permissions="rwxr-x---"
169)
170 
171# Update ACL entries
172from azure.storage.filedatalake import AccessControlChangeResult
173directory_client.update_access_control_recursive(
174    acl="user:user-id:rwx"
175)
176```
177 
178## Async Client
179 
180```python
181from azure.storage.filedatalake.aio import DataLakeServiceClient
182from azure.identity.aio import DefaultAzureCredential
183 
184async def datalake_operations():
185    credential = DefaultAzureCredential()
186    
187    async with DataLakeServiceClient(
188        account_url="https://<account>.dfs.core.windows.net",
189        credential=credential
190    ) as service_client:
191        file_system_client = service_client.get_file_system_client("myfilesystem")
192        file_client = file_system_client.get_file_client("test.txt")
193        
194        await file_client.upload_data(b"async content", overwrite=True)
195        
196        download = await file_client.download_file()
197        content = await download.readall()
198 
199import asyncio
200asyncio.run(datalake_operations())
201```
202 
203## Best Practices
204 
2051. **Use hierarchical namespace** for file system semantics
2062. **Use `append_data` + `flush_data`** for large file uploads
2073. **Set ACLs at directory level** and inherit to children
2084. **Use async client** for high-throughput scenarios
2095. **Use `get_paths` with `recursive=True`** for full directory listing
2106. **Set metadata** for custom file attributes
2117. **Consider Blob API** for simple object storage use cases
212