# act-plus-plus

**Repository Path**: bb188641864/act-plus-plus

## Basic Information

- **Project Name**: act-plus-plus
- **Description**: aloha 神经网络代码
- **Primary Language**: Python
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 1
- **Forks**: 0
- **Created**: 2024-08-01
- **Last Updated**: 2025-01-13

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Mobile ALOHA 的模仿学习算法和协同训练


#### 项目网站: https://mobile-aloha.github.io/
注意：transfer cube 指的是某项功能，有人翻译成动作分块(是不是指得ACT)
This repo contains the implementation of ACT, Diffusion Policy and VINN, together with 2 simulated environments:
Transfer Cube and Bimanual Insertion. You can train and evaluate them in sim or real.
For real, you would also need to install [Mobile ALOHA](https://github.com/MarkFzp/mobile-aloha). This repo is forked from the [ACT repo](https://github.com/tonyzhaozh/act).

该仓库包含 ACT、Diffusion Policy 和 VINN 的实现，以及 2 个模拟环境:Transfer Cube and Bimanual Insertion.您可以模拟或真实地训练和评估它们.
实际上，您还需要安装[Mobile ALOHA](https://github.com/MarkFzp/mobile-aloha).该存储库是从 [ACT repo] 分叉出来的

### Updates:
您可以在以下链接找到模拟环境的所有脚本/人工演示 [here](https://drive.google.com/drive/folders/1gPR03v05S1xiInoVJn7G7VJ9pDCnxq9O?usp=share_link).


### Repo Structure 仓库接口 或关键文件说明
- ``imitate_episodes.py`` Train and Evaluate ACT  训练和评估 ACT
- ``policy.py`` An adaptor for ACT policy  ACT 策略的适配器
- ``detr`` Model definitions of ACT, modified from DETR  ACT 的模型定义，修改自 DETR
- ``sim_env.py`` Mujoco + DM_Control environments with joint space control  具有联合空间控制的 Mujoco + DM_Control 环境
- ``ee_sim_env.py`` Mujoco + DM_Control environments with EE space control 
具有 EE 空间控制的 Mujoco + DM_Control 环境
- ``scripted_policy.py`` Scripted policies for sim environments  模拟环境的脚本化策略
- ``constants.py`` Constants shared across files  跨文件共享的常量
- ``utils.py`` Utils such as data loading and helper functions  数据加载和辅助函数等实用程序
- ``visualize_episodes.py`` Save videos from a .hdf5 dataset  保存 .hdf5 数据集中的视频


### Installation  安装过程

    conda create -n aloha python=3.8.10
    conda activate aloha
    pip install torchvision
    pip install torch
    pip install pyquaternion
    pip install pyyaml
    pip install rospkg
    pip install pexpect
    pip install mujoco==2.3.7
    pip install dm_control==1.0.14
    pip install opencv-python
    pip install matplotlib
    pip install einops
    pip install packaging
    pip install h5py
    pip install ipython
    cd act/detr && pip install -e .

- also need to install（还需要安装） https://github.com/ARISE-Initiative/robomimic/tree/r2d2 (note the r2d2 branch) for Diffusion Policy by `pip install -e .`

- 这步安装比较坑，windows系统下 依赖 egl_probe 安装失败，要换成https://github.com/mhandb/egl_probe/tree/fix_windows_build 这份有源码进行安装。另外需要确保本地安装的visual studio 内包含 cmake,nmake 两个程序(刚开始安装的visual studio 是基础版，没有选择c++开发组件)。另外需要讲cmake,nmake所在目录添加到系统变量如：

    `D:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin(cmake.exe)`

    `D:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\Hostx86\x64\（nmake.exe）`


### Example Usages

To set up a new terminal, run（打开新终端，执行以下操作）:

    conda activate aloha
    cd <path to act repo>

### Simulated experiments(模拟实验) (LEGACY table-top ALOHA environments 传统桌面 ALOHA 环境)

We use ``sim_transfer_cube_scripted`` task in the examples below（我们在下面的示例中使用“sim_transfer_cube_scripted”任务。）. Another option is ``sim_insertion_scripted``（另一个选项是“sim_insertion_scripted”。）.
To generated 50 episodes of scripted data, run:
要生成 50 集脚本数据，请运行：

    python3 record_sim_episodes.py --task_name sim_transfer_cube_scripted --dataset_dir <data save dir> --num_episodes 50

To can add the flag ``--onscreen_render`` to see real-time rendering.可以添加标志“--onscreen_render”来查看实时渲染。
To visualize the simulated episodes after it is collected, run  要在收集后可视化模拟事件，请运行

    python3 visualize_episodes.py --dataset_dir <data save dir> --episode_idx 0

Note: to visualize data from the mobile-aloha hardware, use the visualize_episodes.py from https://github.com/MarkFzp/mobile-aloha

注意：要可视化来自 mobile-aloha 硬件的数据，请使用 Visualize_episodes.py 从 https://github.com/MarkFzp/mobile-aloha


To train ACT（训练ACT）:
    
    # Transfer Cube task
    python3 imitate_episodes.py --task_name sim_transfer_cube_scripted --ckpt_dir <ckpt dir> --policy_class ACT --kl_weight 10 --chunk_size 100 --hidden_dim 512 --batch_size 8 --dim_feedforward 3200 --num_epochs 2000  --lr 1e-5 --seed 0


To evaluate the policy, run the same command but add ``--eval``. This loads the best validation checkpoint.
要评估策略，请运行相同的命令，并加上“--eval”命令。这会加载最佳验证检查点。这会加载最佳验证检查点。

The success rate should be around 90% for transfer cube, and around 50% for insertion.
To enable temporal ensembling, add flag ``--temporal_agg``.
Videos will be saved to ``<ckpt_dir>`` for each rollout.
You can also add ``--onscreen_render`` to see real-time rendering during evaluation.

转移立方体的成功率应在 90% 左右，插入立方体的成功率应在 50% 左右。
要启用时间集成，请添加标志“--temporal_agg”。
每次推出时，视频将保存到“<ckpt_dir>”。
您还可以添加“--onscreen_render”以在评估过程中查看实时渲染。


For real-world data where things can be harder to model, train for at least 5000 epochs or 3-4 times the length after the loss has plateaued.
Please refer to [tuning tips](https://docs.google.com/document/d/1FVIZfoALXg_ZkYKaYVh-qOlaXveq5CtvJHXkY25eYhs/edit?usp=sharing) for more info.

对于难以建模的现实数据，训练至少 5000 个 epoch，或者在损失趋于稳定后训练 3-4 倍的长度。
请参阅[tuning tips](https://docs.google.com/document/d/1FVIZfoALXg_ZkYKaYVh-qOlaXveq5CtvJHXkY25eYhs/edit?usp=sharing)了解更多信息。


### [ACT tuning tips](https://docs.google.com/document/d/1FVIZfoALXg_ZkYKaYVh-qOlaXveq5CtvJHXkY25eYhs/edit?usp=sharing)
TL;DR: if your ACT policy is jerky or pauses in the middle of an episode, just train for longer! Success rate and smoothness can improve way after loss plateaus.
### [ACT 训练技巧](https://docs.google.com/document/d/1FVIZfoALXg_ZkYKaYVh-qOlaXveq5CtvJHXkY25eYhs/edit?usp=sharing)
TL;DR: 如果你得ACT策略不稳定或者一个场景中暂停了，只有更多训练！成功率和平滑度可以在损失达到稳定水平后得到改善。

- Chunk size is the most important param to tune when applying ACT to a new environment. One chunk should correspond to ~1 secs wall-clock robot motion.
- High KL weight (10 or 100), or train without CVAE encoder.
- Consider removing temporal_agg and increase query frequency here to be the same as your chunk size. I.e. each chunk is executed fully.
- train for very long (well after things plateaus, see picture)
- Try to increase batch size as much as possible, and increase lr accordingly. E.g. batch size 64 with learning rate 5e-5 versus batch size 8 and learning rate 1e-5
- Have separate backbones for each camera (requires changing the code, see this commit)
- L1 loss > L2 loss (not precise enough)
- Abs position control > delta/velocity control (harder to recover)
- Try multiple checkpoints

For real-world experiments:
- Train for even longer (5k - 8k steps, especially if multi-camera)
- If inference is too slow -> robot moving slowly: disable temporal_agg and increase query frequency here. We tried as high as 20.


Example loss curve (L1)



- 将 ACT 应用到新环境时，块大小是需要调整的最重要参数。一个块应对应于约 1 秒的挂钟机器人运动。
- 高 KL 权重（10 或 100），或不使用 CVAE 编码器进行训练。
- 考虑删除temporal_agg并增加查询频率，使其与块大小相同。 IE。每个块都被完全执行。
- 训练很长时间（在事情趋于平稳之后，见图片）
- 尝试尽可能增加batch size，并相应增加lr。例如。批量大小 64，学习率 5e-5 与批量大小 8，学习率 1e-5
- 每个相机都有单独的主干（需要更改代码，请参阅此提交）
- L1损失> L2损失（不够精确）
- 绝对位置控制 > 增量/速度控制（更难恢复）
- 尝试多个检查点

对于现实世界的实验：
- 训练时间更长（5k - 8k 步，尤其是多摄像机时）
- 如果推理太慢 -> 机器人移动缓慢：禁用temporal_agg并在此处增加查询频率。我们尝试最高为 20。


损失曲线示例 (L1)

![输入图片说明](assets/image.png)

用官方命令install 版本可控
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118