453 lines
14 KiB
Markdown
453 lines
14 KiB
Markdown
|
|
|||
|
|
|
|||
|
|
A FileSystem Abstraction System for Go
|
|||
|
|
|
|||
|
|
[](https://travis-ci.org/spf13/afero) [](https://ci.appveyor.com/project/spf13/afero) [](https://godoc.org/github.com/spf13/afero) [](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
|
|||
|
|
|
|||
|
|
# Overview
|
|||
|
|
|
|||
|
|
Afero is an filesystem framework providing a simple, uniform and universal API
|
|||
|
|
interacting with any filesystem, as an abstraction layer providing interfaces,
|
|||
|
|
types and methods. Afero has an exceptionally clean interface and simple design
|
|||
|
|
without needless constructors or initialization methods.
|
|||
|
|
|
|||
|
|
Afero is also a library providing a base set of interoperable backend
|
|||
|
|
filesystems that make it easy to work with afero while retaining all the power
|
|||
|
|
and benefit of the os and ioutil packages.
|
|||
|
|
|
|||
|
|
Afero provides significant improvements over using the os package alone, most
|
|||
|
|
notably the ability to create mock and testing filesystems without relying on the disk.
|
|||
|
|
|
|||
|
|
It is suitable for use in a any situation where you would consider using the OS
|
|||
|
|
package as it provides an additional abstraction that makes it easy to use a
|
|||
|
|
memory backed file system during testing. It also adds support for the http
|
|||
|
|
filesystem for full interoperability.
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Afero Features
|
|||
|
|
|
|||
|
|
* A single consistent API for accessing a variety of filesystems
|
|||
|
|
* Interoperation between a variety of file system types
|
|||
|
|
* A set of interfaces to encourage and enforce interoperability between backends
|
|||
|
|
* An atomic cross platform memory backed file system
|
|||
|
|
* Support for compositional (union) file systems by combining multiple file systems acting as one
|
|||
|
|
* Specialized backends which modify existing filesystems (Read Only, Regexp filtered)
|
|||
|
|
* A set of utility functions ported from io, ioutil & hugo to be afero aware
|
|||
|
|
|
|||
|
|
|
|||
|
|
# Using Afero
|
|||
|
|
|
|||
|
|
Afero is easy to use and easier to adopt.
|
|||
|
|
|
|||
|
|
A few different ways you could use Afero:
|
|||
|
|
|
|||
|
|
* Use the interfaces alone to define you own file system.
|
|||
|
|
* Wrap for the OS packages.
|
|||
|
|
* Define different filesystems for different parts of your application.
|
|||
|
|
* Use Afero for mock filesystems while testing
|
|||
|
|
|
|||
|
|
## Step 1: Install Afero
|
|||
|
|
|
|||
|
|
First use go get to install the latest version of the library.
|
|||
|
|
|
|||
|
|
$ go get github.com/spf13/afero
|
|||
|
|
|
|||
|
|
Next include Afero in your application.
|
|||
|
|
```go
|
|||
|
|
import "github.com/spf13/afero"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Step 2: Declare a backend
|
|||
|
|
|
|||
|
|
First define a package variable and set it to a pointer to a filesystem.
|
|||
|
|
```go
|
|||
|
|
var AppFs = afero.NewMemMapFs()
|
|||
|
|
|
|||
|
|
or
|
|||
|
|
|
|||
|
|
var AppFs = afero.NewOsFs()
|
|||
|
|
```
|
|||
|
|
It is important to note that if you repeat the composite literal you
|
|||
|
|
will be using a completely new and isolated filesystem. In the case of
|
|||
|
|
OsFs it will still use the same underlying filesystem but will reduce
|
|||
|
|
the ability to drop in other filesystems as desired.
|
|||
|
|
|
|||
|
|
## Step 3: Use it like you would the OS package
|
|||
|
|
|
|||
|
|
Throughout your application use any function and method like you normally
|
|||
|
|
would.
|
|||
|
|
|
|||
|
|
So if my application before had:
|
|||
|
|
```go
|
|||
|
|
os.Open('/tmp/foo')
|
|||
|
|
```
|
|||
|
|
We would replace it with:
|
|||
|
|
```go
|
|||
|
|
AppFs.Open('/tmp/foo')
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`AppFs` being the variable we defined above.
|
|||
|
|
|
|||
|
|
|
|||
|
|
## List of all available functions
|
|||
|
|
|
|||
|
|
File System Methods Available:
|
|||
|
|
```go
|
|||
|
|
Chmod(name string, mode os.FileMode) : error
|
|||
|
|
Chtimes(name string, atime time.Time, mtime time.Time) : error
|
|||
|
|
Create(name string) : File, error
|
|||
|
|
Mkdir(name string, perm os.FileMode) : error
|
|||
|
|
MkdirAll(path string, perm os.FileMode) : error
|
|||
|
|
Name() : string
|
|||
|
|
Open(name string) : File, error
|
|||
|
|
OpenFile(name string, flag int, perm os.FileMode) : File, error
|
|||
|
|
Remove(name string) : error
|
|||
|
|
RemoveAll(path string) : error
|
|||
|
|
Rename(oldname, newname string) : error
|
|||
|
|
Stat(name string) : os.FileInfo, error
|
|||
|
|
```
|
|||
|
|
File Interfaces and Methods Available:
|
|||
|
|
```go
|
|||
|
|
io.Closer
|
|||
|
|
io.Reader
|
|||
|
|
io.ReaderAt
|
|||
|
|
io.Seeker
|
|||
|
|
io.Writer
|
|||
|
|
io.WriterAt
|
|||
|
|
|
|||
|
|
Name() : string
|
|||
|
|
Readdir(count int) : []os.FileInfo, error
|
|||
|
|
Readdirnames(n int) : []string, error
|
|||
|
|
Stat() : os.FileInfo, error
|
|||
|
|
Sync() : error
|
|||
|
|
Truncate(size int64) : error
|
|||
|
|
WriteString(s string) : ret int, err error
|
|||
|
|
```
|
|||
|
|
In some applications it may make sense to define a new package that
|
|||
|
|
simply exports the file system variable for easy access from anywhere.
|
|||
|
|
|
|||
|
|
## Using Afero's utility functions
|
|||
|
|
|
|||
|
|
Afero provides a set of functions to make it easier to use the underlying file systems.
|
|||
|
|
These functions have been primarily ported from io & ioutil with some developed for Hugo.
|
|||
|
|
|
|||
|
|
The afero utilities support all afero compatible backends.
|
|||
|
|
|
|||
|
|
The list of utilities includes:
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
DirExists(path string) (bool, error)
|
|||
|
|
Exists(path string) (bool, error)
|
|||
|
|
FileContainsBytes(filename string, subslice []byte) (bool, error)
|
|||
|
|
GetTempDir(subPath string) string
|
|||
|
|
IsDir(path string) (bool, error)
|
|||
|
|
IsEmpty(path string) (bool, error)
|
|||
|
|
ReadDir(dirname string) ([]os.FileInfo, error)
|
|||
|
|
ReadFile(filename string) ([]byte, error)
|
|||
|
|
SafeWriteReader(path string, r io.Reader) (err error)
|
|||
|
|
TempDir(dir, prefix string) (name string, err error)
|
|||
|
|
TempFile(dir, prefix string) (f File, err error)
|
|||
|
|
Walk(root string, walkFn filepath.WalkFunc) error
|
|||
|
|
WriteFile(filename string, data []byte, perm os.FileMode) error
|
|||
|
|
WriteReader(path string, r io.Reader) (err error)
|
|||
|
|
```
|
|||
|
|
For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero)
|
|||
|
|
|
|||
|
|
They are available under two different approaches to use. You can either call
|
|||
|
|
them directly where the first parameter of each function will be the file
|
|||
|
|
system, or you can declare a new `Afero`, a custom type used to bind these
|
|||
|
|
functions as methods to a given filesystem.
|
|||
|
|
|
|||
|
|
### Calling utilities directly
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
fs := new(afero.MemMapFs)
|
|||
|
|
f, err := afero.TempFile(fs,"", "ioutil-test")
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Calling via Afero
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
fs := afero.NewMemMapFs()
|
|||
|
|
afs := &afero.Afero{Fs: fs}
|
|||
|
|
f, err := afs.TempFile("", "ioutil-test")
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Using Afero for Testing
|
|||
|
|
|
|||
|
|
There is a large benefit to using a mock filesystem for testing. It has a
|
|||
|
|
completely blank state every time it is initialized and can be easily
|
|||
|
|
reproducible regardless of OS. You could create files to your heart’s content
|
|||
|
|
and the file access would be fast while also saving you from all the annoying
|
|||
|
|
issues with deleting temporary files, Windows file locking, etc. The MemMapFs
|
|||
|
|
backend is perfect for testing.
|
|||
|
|
|
|||
|
|
* Much faster than performing I/O operations on disk
|
|||
|
|
* Avoid security issues and permissions
|
|||
|
|
* Far more control. 'rm -rf /' with confidence
|
|||
|
|
* Test setup is far more easier to do
|
|||
|
|
* No test cleanup needed
|
|||
|
|
|
|||
|
|
One way to accomplish this is to define a variable as mentioned above.
|
|||
|
|
In your application this will be set to afero.NewOsFs() during testing you
|
|||
|
|
can set it to afero.NewMemMapFs().
|
|||
|
|
|
|||
|
|
It wouldn't be uncommon to have each test initialize a blank slate memory
|
|||
|
|
backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere
|
|||
|
|
appropriate in my application code. This approach ensures that Tests are order
|
|||
|
|
independent, with no test relying on the state left by an earlier test.
|
|||
|
|
|
|||
|
|
Then in my tests I would initialize a new MemMapFs for each test:
|
|||
|
|
```go
|
|||
|
|
func TestExist(t *testing.T) {
|
|||
|
|
appFS := afero.NewMemMapFs()
|
|||
|
|
// create test files and directories
|
|||
|
|
appFS.MkdirAll("src/a", 0755)
|
|||
|
|
afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644)
|
|||
|
|
afero.WriteFile(appFS, "src/c", []byte("file c"), 0644)
|
|||
|
|
name := "src/c"
|
|||
|
|
_, err := appFS.Stat(name)
|
|||
|
|
if os.IsNotExist(err) {
|
|||
|
|
t.Errorf("file \"%s\" does not exist.\n", name)
|
|||
|
|
}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
# Available Backends
|
|||
|
|
|
|||
|
|
## Operating System Native
|
|||
|
|
|
|||
|
|
### OsFs
|
|||
|
|
|
|||
|
|
The first is simply a wrapper around the native OS calls. This makes it
|
|||
|
|
very easy to use as all of the calls are the same as the existing OS
|
|||
|
|
calls. It also makes it trivial to have your code use the OS during
|
|||
|
|
operation and a mock filesystem during testing or as needed.
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
appfs := afero.NewOsFs()
|
|||
|
|
appfs.MkdirAll("src/a", 0755))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Memory Backed Storage
|
|||
|
|
|
|||
|
|
### MemMapFs
|
|||
|
|
|
|||
|
|
Afero also provides a fully atomic memory backed filesystem perfect for use in
|
|||
|
|
mocking and to speed up unnecessary disk io when persistence isn’t
|
|||
|
|
necessary. It is fully concurrent and will work within go routines
|
|||
|
|
safely.
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
mm := afero.NewMemMapFs()
|
|||
|
|
mm.MkdirAll("src/a", 0755))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
#### InMemoryFile
|
|||
|
|
|
|||
|
|
As part of MemMapFs, Afero also provides an atomic, fully concurrent memory
|
|||
|
|
backed file implementation. This can be used in other memory backed file
|
|||
|
|
systems with ease. Plans are to add a radix tree memory stored file
|
|||
|
|
system using InMemoryFile.
|
|||
|
|
|
|||
|
|
## Network Interfaces
|
|||
|
|
|
|||
|
|
### SftpFs
|
|||
|
|
|
|||
|
|
Afero has experimental support for secure file transfer protocol (sftp). Which can
|
|||
|
|
be used to perform file operations over a encrypted channel.
|
|||
|
|
|
|||
|
|
## Filtering Backends
|
|||
|
|
|
|||
|
|
### BasePathFs
|
|||
|
|
|
|||
|
|
The BasePathFs restricts all operations to a given path within an Fs.
|
|||
|
|
The given file name to the operations on this Fs will be prepended with
|
|||
|
|
the base path before calling the source Fs.
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path")
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### ReadOnlyFs
|
|||
|
|
|
|||
|
|
A thin wrapper around the source Fs providing a read only view.
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
fs := afero.NewReadOnlyFs(afero.NewOsFs())
|
|||
|
|
_, err := fs.Create("/file.txt")
|
|||
|
|
// err = syscall.EPERM
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
# RegexpFs
|
|||
|
|
|
|||
|
|
A filtered view on file names, any file NOT matching
|
|||
|
|
the passed regexp will be treated as non-existing.
|
|||
|
|
Files not matching the regexp provided will not be created.
|
|||
|
|
Directories are not filtered.
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`))
|
|||
|
|
_, err := fs.Create("/file.html")
|
|||
|
|
// err = syscall.ENOENT
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### HttpFs
|
|||
|
|
|
|||
|
|
Afero provides an http compatible backend which can wrap any of the existing
|
|||
|
|
backends.
|
|||
|
|
|
|||
|
|
The Http package requires a slightly specific version of Open which
|
|||
|
|
returns an http.File type.
|
|||
|
|
|
|||
|
|
Afero provides an httpFs file system which satisfies this requirement.
|
|||
|
|
Any Afero FileSystem can be used as an httpFs.
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
httpFs := afero.NewHttpFs(<ExistingFS>)
|
|||
|
|
fileserver := http.FileServer(httpFs.Dir(<PATH>)))
|
|||
|
|
http.Handle("/", fileserver)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Composite Backends
|
|||
|
|
|
|||
|
|
Afero provides the ability have two filesystems (or more) act as a single
|
|||
|
|
file system.
|
|||
|
|
|
|||
|
|
### CacheOnReadFs
|
|||
|
|
|
|||
|
|
The CacheOnReadFs will lazily make copies of any accessed files from the base
|
|||
|
|
layer into the overlay. Subsequent reads will be pulled from the overlay
|
|||
|
|
directly permitting the request is within the cache duration of when it was
|
|||
|
|
created in the overlay.
|
|||
|
|
|
|||
|
|
If the base filesystem is writeable, any changes to files will be
|
|||
|
|
done first to the base, then to the overlay layer. Write calls to open file
|
|||
|
|
handles like `Write()` or `Truncate()` to the overlay first.
|
|||
|
|
|
|||
|
|
To writing files to the overlay only, you can use the overlay Fs directly (not
|
|||
|
|
via the union Fs).
|
|||
|
|
|
|||
|
|
Cache files in the layer for the given time.Duration, a cache duration of 0
|
|||
|
|
means "forever" meaning the file will not be re-requested from the base ever.
|
|||
|
|
|
|||
|
|
A read-only base will make the overlay also read-only but still copy files
|
|||
|
|
from the base to the overlay when they're not present (or outdated) in the
|
|||
|
|
caching layer.
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
base := afero.NewOsFs()
|
|||
|
|
layer := afero.NewMemMapFs()
|
|||
|
|
ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### CopyOnWriteFs()
|
|||
|
|
|
|||
|
|
The CopyOnWriteFs is a read only base file system with a potentially
|
|||
|
|
writeable layer on top.
|
|||
|
|
|
|||
|
|
Read operations will first look in the overlay and if not found there, will
|
|||
|
|
serve the file from the base.
|
|||
|
|
|
|||
|
|
Changes to the file system will only be made in the overlay.
|
|||
|
|
|
|||
|
|
Any attempt to modify a file found only in the base will copy the file to the
|
|||
|
|
overlay layer before modification (including opening a file with a writable
|
|||
|
|
handle).
|
|||
|
|
|
|||
|
|
Removing and Renaming files present only in the base layer is not currently
|
|||
|
|
permitted. If a file is present in the base layer and the overlay, only the
|
|||
|
|
overlay will be removed/renamed.
|
|||
|
|
|
|||
|
|
```go
|
|||
|
|
base := afero.NewOsFs()
|
|||
|
|
roBase := afero.NewReadOnlyFs(base)
|
|||
|
|
ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs())
|
|||
|
|
|
|||
|
|
fh, _ = ufs.Create("/home/test/file2.txt")
|
|||
|
|
fh.WriteString("This is a test")
|
|||
|
|
fh.Close()
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
In this example all write operations will only occur in memory (MemMapFs)
|
|||
|
|
leaving the base filesystem (OsFs) untouched.
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Desired/possible backends
|
|||
|
|
|
|||
|
|
The following is a short list of possible backends we hope someone will
|
|||
|
|
implement:
|
|||
|
|
|
|||
|
|
* SSH
|
|||
|
|
* ZIP
|
|||
|
|
* TAR
|
|||
|
|
* S3
|
|||
|
|
|
|||
|
|
# About the project
|
|||
|
|
|
|||
|
|
## What's in the name
|
|||
|
|
|
|||
|
|
Afero comes from the latin roots Ad-Facere.
|
|||
|
|
|
|||
|
|
**"Ad"** is a prefix meaning "to".
|
|||
|
|
|
|||
|
|
**"Facere"** is a form of the root "faciō" making "make or do".
|
|||
|
|
|
|||
|
|
The literal meaning of afero is "to make" or "to do" which seems very fitting
|
|||
|
|
for a library that allows one to make files and directories and do things with them.
|
|||
|
|
|
|||
|
|
The English word that shares the same roots as Afero is "affair". Affair shares
|
|||
|
|
the same concept but as a noun it means "something that is made or done" or "an
|
|||
|
|
object of a particular type".
|
|||
|
|
|
|||
|
|
It's also nice that unlike some of my other libraries (hugo, cobra, viper) it
|
|||
|
|
Googles very well.
|
|||
|
|
|
|||
|
|
## Release Notes
|
|||
|
|
|
|||
|
|
* **0.10.0** 2015.12.10
|
|||
|
|
* Full compatibility with Windows
|
|||
|
|
* Introduction of afero utilities
|
|||
|
|
* Test suite rewritten to work cross platform
|
|||
|
|
* Normalize paths for MemMapFs
|
|||
|
|
* Adding Sync to the file interface
|
|||
|
|
* **Breaking Change** Walk and ReadDir have changed parameter order
|
|||
|
|
* Moving types used by MemMapFs to a subpackage
|
|||
|
|
* General bugfixes and improvements
|
|||
|
|
* **0.9.0** 2015.11.05
|
|||
|
|
* New Walk function similar to filepath.Walk
|
|||
|
|
* MemMapFs.OpenFile handles O_CREATE, O_APPEND, O_TRUNC
|
|||
|
|
* MemMapFs.Remove now really deletes the file
|
|||
|
|
* InMemoryFile.Readdir and Readdirnames work correctly
|
|||
|
|
* InMemoryFile functions lock it for concurrent access
|
|||
|
|
* Test suite improvements
|
|||
|
|
* **0.8.0** 2014.10.28
|
|||
|
|
* First public version
|
|||
|
|
* Interfaces feel ready for people to build using
|
|||
|
|
* Interfaces satisfy all known uses
|
|||
|
|
* MemMapFs passes the majority of the OS test suite
|
|||
|
|
* OsFs passes the majority of the OS test suite
|
|||
|
|
|
|||
|
|
## Contributing
|
|||
|
|
|
|||
|
|
1. Fork it
|
|||
|
|
2. Create your feature branch (`git checkout -b my-new-feature`)
|
|||
|
|
3. Commit your changes (`git commit -am 'Add some feature'`)
|
|||
|
|
4. Push to the branch (`git push origin my-new-feature`)
|
|||
|
|
5. Create new Pull Request
|
|||
|
|
|
|||
|
|
## Contributors
|
|||
|
|
|
|||
|
|
Names in no particular order:
|
|||
|
|
|
|||
|
|
* [spf13](https://github.com/spf13)
|
|||
|
|
* [jaqx0r](https://github.com/jaqx0r)
|
|||
|
|
* [mbertschler](https://github.com/mbertschler)
|
|||
|
|
* [xor-gate](https://github.com/xor-gate)
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Afero is released under the Apache 2.0 license. See
|
|||
|
|
[LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt)
|